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ABSTRACT 


This  thesis  evaluated  the  first  iteration  of  Navy  Postgraduate  School  (NPS)  on¬ 
line  course  ‘Space  Systems  -  Technology  and  Applications’.  The  thesis  goal  was  to 
evaluate:  course  design;  course  effectiveness;  student  expectations  and  motivation  before 
and  affective  reactions  after  the  course;  and  typical  patterns  of  on-line  learning  behavior 
according  to  Long’s  theory.  Ultimately,  the  thesis  intended  to  derive  lessons  learned  and 
recommendations  for  future  distributed  learning  (DL)  courses  and  future  research  in  this 
area. 

There  were  thirty  regular,  resident  NPS  students  in  the  sample.  Data  was  collected 
from  nine  sources:  four  on-line  questionnaires,  Student  Opinion  Forms,  on-line  tracking, 
final  grades,  interviews  with  students,  and  interviews  with  the  instructor. 

Even  though  the  course  first  course  taught  by  this  instructor,  it  was  successful. 
Student  expectations,  motivation,  affective  and  utility  reactions  were  predominantly 
positive.  For  students,  the  most  important  advantage  of  the  course  was  time  flexibility 
and  convenience.  Students  and  the  instructor  agreed  about  central  role  of 
interactivity/feedback  in  a  DL,  which  should  be  included  into  models  of  training 
effectiveness  evaluation.  Long-Dziuban’s  protocol  results  were  surprising,  but  solid 
conclusions  cannot  be  made  without  focused  research.  Results  provided  several 
recommendations  about  the  course  design,  pedagogical  improvements,  instructional 
counseling,  and  future  research. 
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I.  INTRODUCTION 


This  chapter  provides  the  reader  with  the  background  that  led  to  the  creation  of 
this  thesis,  outlines  the  theoretical  models  that  guided  and  directed  the  research,  supplies 
the  thesis  purpose  and  the  research  questions,  briefly  reviews  the  thesis  scope  and 
methodology,  and  provides  an  overview  on  the  organization  of  the  study. 

This  thesis  tests  a  model  of  a  web-based  course  evaluation,  based  on  specific 
theoretical  bases,  which  are  described  later  in  this  chapter  and  in  Chapter  II.  This  model 
determines  the  research  questions  of  this  thesis,  its  methodology,  and  the  study’s 
organization. 

My  intention  is  to  evaluate  die  utility  of  the  model  through  the  qualitative  and 
quantitative  analysis  of  data  collected  in  this  case  study  research  in  the  context  of 
relationships  and  factors  that  this  model  would  predict.  Lessons  learned  from  the  case 
study  should  primarily  have  practical  applicability.  However,  the  study  might  also 
provide  conclusions  in  terms  of  known  models  and  their  possible  modifications  and 
improvements. 

A.  BACKGROUND 

This  section  provides  a  brief  description  of  events,  circumstances,  and  theoretical 
concepts  that  explain  the  need  for  this  research. 

Several  years  ago,  the  Chief  of  Naval  Education  and  Training  (CNET),  U.S. 
Navy,  intending  to  enhance  education  and  training  opportunities  for  Navy  personnel  and 
being  aware  of  an  increasing  importance  of  distributed  learning  (DL)  in  contemporary 
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education  and  training,  initiated  an  idea  for  a  pilot  project.  The  project  did  not  have  an 
official  name,  but  in  fiscal  year  1999  all  stakeholders  agreed  to  the  project  and  CNET 
ensured  its  funding.  Decision-makers  concluded  that  the  Naval  Postgraduate  School 
(NPS)  was  an  excellent  place  to  run  the  pilot  program.  NPS  agreed  to  transform  several 
of  its  courses  in  the  Information  Systems  and  Operations  (ISO)  curriculum ^  into  web- 
based,  on-line  courses  and  to  delivere  them.  In  the  first  phase,  the  program  would  include 
two  courses,  one  of  which  was  ‘Space  Systems  -  Technology  and  Applications’  taught  in 
the  Space  Systems  Academic  Group.  In  total,  the  plan  was  to  gradually  transform  twelve 
to  fourteen  courses  to  DL  versions.  By  the  end  of  2000  four  courses  were  on-line,  and  six 
more  were  planned  for  2001. 

The  reasons  for  such  broad  distributed  learning  initiatives  in  the  military  are 
understandable.  The  modem  military  organization  is  very  information-oriented  and 
highly  training-demanding  because  of  the  rapid  development  of  military  technologies. 
The  vision  for  future  battlefields  is  interactive  and  network-centered  with 
communications  playing  a  crucial  role.  Distributed  learning  is  one  of  the  inevitable, 
powerful  training  tools  the  military  needs  to  obtain  such  a  vision.  During  the  last  10 
years,  U.S.  and  foreign  universities,  other  educational  institutions,  and  companies  in 
general  have  instituted  DL  courses  and  programs.  Progress  in  this  instructional  area  hag 
been  very  rapid. 

Before  going  further,  I  have  to  precisely  define  the  central  theme  in  this  thesis: 
distributed  learning.  Bates  (1999),  in  his  excellent  book  ‘Managing  Technological 
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Change’  (p.  27),  quotes  the  definition  of  distributed  learning  by  the  Institute  for 
Academic  Technology,  University  of  North  Carolina: 

A  distributed  learning  environment  is  a  learner-centered  approach  to 
education,  which  integrates  a  number  of  technologies  to  enable 
opportunities  for  activities  and  interaction  in  both  asynchronous  and  real¬ 
time  modes.  The  model  is  based  on  blending  a  choice  of  appropriate 
technologies  with  aspects  of  campus-based  delivery,  open  learning 
systems,  and  distance  education.  The  approach  gives  instructors  the 
flexibility  to  customize  learning  environments  to  meet  the  needs  of  diverse 
students  populations,  while  providing  both  high-quality  and  cost-effective 
learning. 

Later  in  his  text.  Bates  (1999)  continues  elaborating  on  this  terminology  (p.  28): 

New  technologies  are  leading  to  major  structural  changes  in  the 
management  and  organization  of  teaching.  These  developments  are 
increasingly  being  referred  to  in  the  United  States  and  Canada  as 
distributed  learning,  in  the  United  Kingdom  as  networked  learning,  and  in 
Australia  as  flexible  learning. 

Belanger  and  Jordan  (2000,  pp.  49-53,  71-86)  divide  computer-aided  instruction 
(CAI)  into  four  main  groups: 

•  Web-based  training  (WBT) 

•  Teleconferencing 

•  Videotape  instructions 

•  Video  tele  training  (VTT) 

This  thesis  is  focused  on  WBT.  In  Belanger  and  Jordan  WBT,  with  the  Internet  as 
a  primary  medium  for  delivery  of  instruction,  is  then  divided  into: 

•  Web  Enhanced  Courses  -  traditional  classroom  combined  with  Web 
technologies 

•  Web  Managed  Courses  —  where  all  course  management,  testing, 
collaboration,  instructional  material,  etc.  are  on-line 

•  Web  Delivered  Courses  -  all  materials  available  through  the  Web, 
including  tools  for  real-time  collaboration  and  interactivity 

•  Hybrid  Delivery  -  partial  Web  and  partial  CD-ROM  course  delivery 
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This  conceptual  division  of  WBT  still  does  not  provide  a  sufficiently  precise 
definition.  Williams,  Paprock,  and  Covington  (1999)  provide  a  developmental  continuum 
of  distance  learning,  where  the  third  level  of  development  is  closest  to  the  term 
‘distributed  learning’  (Figure  1). 


Level  3  (1990s  to  21st 

Century) 

Highly  interactive 

^  Digital  TV/Multitasking  systems 
Software/TV  network  collaboration 

I  Multimedia  CBT/CD-ROM/laser  disk 

Hybrid  networks 

#  Intemet/WEB/on-line  video 

Virtual  classrooms 

Business,  home,  travel,  continual 

/ 

Level  2  (1960s  to  1990s)  / 

Passive  to  moderately  active  y 

Level  1  (1880s  to  * 

1960s)  Passive  ^ 

Two-way  mteractive  audxo/video 

PC  teletraining  via  the  bulletin  board  system 
Electronic  mail,  computer-mediated 
conferences  (CMCs) 

Computer-based  training  (CBT) 

Satellite  video  teletraining  (one-way 

video/audio) 

Audio/Videotape 

Audio  teletraining  (two-way  audio) 

Radio 

Correspondence 

Telematics  (audiographics) 

Figure  1.  Distance  Education  Continuum  [From:  Williams,  Paprock,  and  Covington 

(1999)]. 

This  continuum  shows  the  way  multi-media  based  learning  has  evolved  through 

the  years,  highlighting  the  development  of  interactive  delivery  environments.  Level  1,  die 

most  passive  form  of  distance  learning,  consists  of  printed  material,  audio/videotapes, 

and  radio  transmission.  This  type  of  learning  is  asynchronous  because  the  instructor  and 

student  transmit  messages  only  one-way  and  receive  responses,  after  a  lengthy  delay. 

Level  2  is  passive  to  moderately  active,  and  includes  two-way  audio  training,  one-way 

video/two-way  audio  tele-training,  computer-based  training  (CBT),  CD-ROMs,  laser 

disk,  e-mail,  computer-mediated  conferences,  audio  graphics,  laser  disks,  and  other 

similar  media.  It  is  considered  synchronous.  Level  3  is  a  highly  interactive  learning 
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environment,  and  the  content  of  a  course  determines  which  technologies  would  be  used 
as  a  primary  instructional  form  of  delivery.  At  this  level,  hybrid  environments  and  all 
distributed  learning  technologies  mentioned  above,  including  the  Internet  and  World 
Wide  Web,  are  combined  into  a  ‘virtual  classroom’. 

Moskal  and  Dziuban  (2000,  p.  2)  quote  Moore  and  Kearsley’s  definition  of 
distance  education: 

[Distance  education  is]  planned  learning  that  normally  occurs  in  a 
different  place  from  teaching  and  as  a  result  requires  special  techniques  of 
course  design,  special  instructional  techniques,  special  methods  of 
communication  by  electronic  or  other  technology,  as  well  as  special 
organizational  and  administrative  arrangements. 

In  the  same  text  (p.  3),  Moskal  and  Dziuban  added  an  important  note:  ‘University 
of  Central  Florida  (UCF)  uses  the  term  distributed  learning  because  courses  offered 
involve  distributing  instruction  using  various  media  to  students  that  may  not,  by 
definition,  be  at  a  distance’.  In  other  words,  DL,  as  defined  by  UCF,  is  mostly  web-based 
and  on-line,  but  may  include  videotape,  two-way  interaction  television,  and  other  similar 
media 


Hartman,  Dziuban,  and  Moskal  (1999,  p.  2)  further  explain  how  the  University  of 
Central  Florida  defines  and  divides  asynchronous  learning  networks  (ALN),  which  is 
very  close  to  the  term  ‘distributed  learning’  used  in  this  thesis: 

ALN  activity  at  UCF  is  characterized  as  one  of  three  types.  Web-enhanced 
(‘E’)  courses  are  those  using  the  Web  in  a  useful  but  incidental  manner. 
Typically,  on-line  materials  are  provided  as  supplemental  or  enrichment 
resources.  All  ‘E’  courses  use  WebCT  as  a  repository  for  course  materials, 
resource  links,  or  for  conferencing  and  e-mail.  Media-enhanced  (‘M’) 
courses  make  significant  use  of  the  Web  as  an  integral  element  of  course 
delivery.  Currently,  some  ‘M’  courses  have  a  reduced  number  of  class 
meetings  (reduced  seat  time),  using  ALN  activities  to  replace  traditional 
class  meetings.  Fully-based  courses  are  designated  ‘W’,  and  have  no 


5 


required  class  meetings.  Some  have  an  optional  pre-course  orientation 

and/or  required  final  examination. 

This  definition  is  much  more  appropriate  for  the  needs  of  this  thesis,  because  the 
course  this  diesis  examines  was  designed  and  delivered  based  on  the  “UCF  ‘W*  course 
model”.  From  here  on,  by  DL  (unless  stated  differently)  I  mean  asynchronous,  web- 
based  course/training,  but  with  the  small  possibility  of  some  non  -  on-line  content.  The 
course  this  thesis  assesses  is  closest  to  the  ‘W’  model  stated  above.  By  synchronicity  I 
mean  that  time  must  be  scheduled  for  the  on-line  meetings  to  be  held,  with  all 
participants  available  at  that  time. 

B.  ON-LINE  EDUCATION 

Students  of  all  ages  and  instructors  worldwide  are  witnessing  an  explosion  of  on¬ 
line  education  -  evidence  of  this  statement  is  visible  in  mass  media,  campuses,  and  in 
everyday  life.  The  best  example  is  the  British  Open  University  (http  ://www.open.ac.uk~). 
the  world’s  largest  and  most  successful  DL  institution. 

The  following  examples  are  listed  in  Kearsley’s  ‘Online  Education’  (2000).  One 
of  the  U.S.  leaders  working  with  schools  and  creating  worthwhile  learning  programs  is 
NASA  (http://www.nasa.gov~).  which  provides  exciting  help  in  its  subject  matter  to 
school  children  and  adults.  Companies  heavily  involved  in  school  networking  are,  for 
instance,  Pacific  Bell  (http://www.kn.pacbell.com~)  and  Hewlett-Packard 
(http://www.hp.com~). 

It  is  impossible  to  list  all  of  the  other  good  examples  of  specific  areas  in  on-line 
learning.  There  are  sites  for  the  senior  learning  community  (http://www.seniomet.org~). 
for  the  disabled  (http://www/rit.edu/~easi~>.  for  teachers  (http://www.webteacher.org~).  and 
even  about  how  to  leam  on-line  (http://www.mtsu.edu/~studskl  or 
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http://wvyw.fami1veducation.comy  Also,  there  are  paper  and  cyber  periodicals  and 
magazines  devoted  to  on-line  education,  like  ‘Syllabus’  (see  http://www.svllabus.comy 
and  numerous  conferences  every  year. 

The  web-based  version  of  SS3011  ‘Space  Systems  -  Technology  and 
Applications’  served  as  a  pilot-course  and  a  beta  test  to  evaluate  if  course  material  could 
be  delivered  effectively  via  the  Internet.  Accordingly,  the  specific  intention  of  the  thesis 
is  to  examine  the  course,  to  evaluate  its  outcomes,  to  analyze  how  effective  it  was  in 
terms  of  meeting  course  objectives,  and  to  identify  the  problems,  file  obstacles,  and  other 
major  factors,  which  played  significant  roles  in  the  effectiveness  of  the  course. 

The  course  was  the  first  designed  and  taught  by  the  instructor,  Commander  Susan 
Higgins.  It  was  the  first  on-line  web-based  course  prepared  and  delivered  in  the  Space 
Systems  Academic  Group  at  NPS.  Prior  to  teaching  the  course,  the  instructor  participated 
in  a  faculty  training  course  entitled  ‘Interactive  Distributed  Learning  for  Technology- 
Mediated  Course  Delivery’  (IDL  6543),  at  the  University  of  Central  Florida,  Orlando, 
FL. 

UCF  began  training  faculty  to  design  and  develop  on-line  courses  in  June  1996, 

and  over  150  UCF  faculty  have  attended  the  IDL  6543  course.  The  Course  Development 

and  Web  Services  Department  at  UCF  not  only  helps  faculty  in  integrating  technology 

and  media  to  transform  the  learning  process  but  also  provides  multimedia  courseware, 

software  and  databases,  learner  support  for  success  in  technology-mediated  classes, 

support  web  site  development  including  the  University’s  primary  site,  and  research  and 

develop  advanced  technological  applications  (Barbara  Truman-Davis:  Brochure  on 

Course  Development  &  Web  Services,  UCF  [2000]).  The  IDL  course’s  site  is  located  at 
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http ://reach.ucf.edu/~idl6543 .  learner  support  at  http://reach.ucf.edii/~coursdev/leaming 
and  http://reach.ucf.edu/~coursedev/cdrom.  and  information  about  WebCT,  UCF  web- 
based  course  management  tool  is  at  http://reach.ucf.edu/~coursedev/webct/sunport.html 
(UCF  [2000]).  More  information  about  WebCT  can  be  found  at  http://www.webct.com 
(UCF  [1999]). 

The  ‘UCF  model’  of  distributed  learning  is  well  known  and  recognized  nationally 
and  internationally.  UCF  received  an  APOQ-SHEEO  (American  Productivity  and  Quality 
Center  and  the  State  Higher  Education  Executive  Officers)  faculty  development  award  in 
1998.  The  United  States  Distance  Learning  Association  (USDLA)  awarded  UCF  in  1999 
the  ‘Excellence  in  Distance  Learning  Programming  Award  in  Higher  Education  for  its 
educational  use  of  distance  learning  media,  including  the  World  Wide  Web,  interactive 
television,  radio,  and  TV  tapes.  Also,  in  1999  it  was  granted  $200,000  from  the  Pew 
Learning  and  Technology  Program  at  the  Center  for  Academic  Transformation  as  part  of 
the  Pew  Grand  Program  in  Course  Redesign. 

C.  PURPOSE 

This  thesis  evaluates  the  SS3011  course  ‘Space  Systems  -  Technology  and 
Applications’  taught  during  the  summer  quarter  of  academic  year  2000  at  NPS.  My  goal 
was  to  employ  a  case  study  approach  that  embodied  components  of  a  theoretical  model  of 
web-based  learning  effectiveness. 

The  research  evaluates  the  following: 

•  Course  design  -  course  objectives,  content,  and  course  activities  as  well  as 
pedagogical  strategies  used  in  that  design. 

•  Course  effectiveness  in  terms  of  student  learning. 

•  Student  expectations  and  motivation  before  the  course  and  affective 
reactions  after  taking  the  course 
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•  Typical  patterns  of  online  learning  behavior. 

•  Student  confidence  in  applying  skills  and  competencies  in  future  jobs 

Most  importantly,  this  research  integrates  the  above-mentioned  variables  into  an 

interactive  system  so  as  to  understand  better  the  influences  of  these  factors  on  each  part 
of  the  learning  process  in  the  course. 

D.  RESEARCH  QUESTIONS 

Primary  research  questions  include  the  following: 

•  How  successful  was  the  web-based  SS3011  course  design  in  meeting 
learning  outcomes? 

•  What  were  the  students’  expectations,  motivation,  on-line  behavior 
patterns,  cognitive  outcomes,  affective-type  reactions,  and  utility-type 
reactions  during  and  after  the  course? 

•  What  recommendations  for  course  improvement,  and  what  lessons  learned 
(in  general)  could  be  derived  from  the  case  analysis  of  this  course? 

Secondary  research  questions: 

•  How  effective  were  the  course  organization,  pedagogy,  and  technology  in 
meeting  course  objectives? 

•  What  are  recommendations  for  future  evaluations  and  research  on  this 
course  and  other  DL  courses  in  future? 


E.  METHODOLOGY  REMARKS 

The  process  of  collecting  data  necessary  for  this  thesis  was  shaped  by  the 
following  factors: 

•  The  ongoing  course  was  the  primary  priority.  The  research  required 
limited  interference  with  the  learning  process,  i.e.  not  disturbing  either  the 
instructor  or  the  students. 

•  Students  voluntarily  cooperated  in  the  research.  If  any  student,  for 
whatever  reason,  refused  to  participate  at  any  time  during  the  course,  that 
decision  was  honored.  None  of  the  students  refused  participation,  but 
some  skipped  some  of  the  questionnaires  and  the  interview.  This 
phenomenon  of  students  skipping  or  not  giving  feedback  is  an  interesting 
issue  itself  and  will  be  addressed  in  other  chapters. 
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•  I  was  obligated  to  guarantee  the  protection  of  the  students’  identity  and 
proper  use  of  all  collected  data.  In  accordance  with  this  element  of  the 
research,  I  do  not  mention  student  names  in  the  thesis. 

•  The  number  of  students  was  not  large  -  thirty-one  students  successfully 
finished  the  course.  Consequently,  this  thesis  uses  the  case  study  approach 
with  the  intention  of  doing  qualitative  and  quantitative  analysis. 

•  The  sample  had  to  be  limited  only  to  students  who  met  following 
important  characteristics:  they  were  regular  students  of  the  course  and 
later  had  successfully  completed  it,  and  they  were  resident  students  at 
NPS. 


There  were  four  main  sources  of  data  relevant  for  this  case  study: 

•  Demographic  data,  student  biographies,  their  grades,  all  student  activity 
tracking  data  collected  in  software  used  in  the  course  (WebCT),  and 
Student  Opinion  Forms  (SOFs). 

•  Evaluation  questionnaires  put  on-line  only  for  the  purpose  of  this  thesis. 
There  were  four  such  instruments  (Appendices  A  through  D). 

•  Individual  semi-structured  interviews  I  conducted  with  17  students  after 
the  course  was  over  (Appendix  E). 

•  Extensive  interviews  I  conducted  with  the  instructor,  mostly  after  the 
course 

Detailed  illustrations  and  discussions  on  all  these  sources  and  data  are  in  Chapter 
III  (Methodology)  and  in  Chapter  IV  (Results  and  Data  Analysis). 

F.  ORGANIZATION  OF  THE  THESIS 

I  review  the  relevant  literature  in  Chapter  n.  Chapter  HI  deals  with  the 

methodology  and  data  collecting  issues.  Chapter  IV  describes  and  analyzes  all  collected 
data.  It  is  followed  by  a  discussion  and  analysis  of  the  results  and  their  implications  in 
Chapter  V,  and  by  conclusions  and  recommendations  in  Chapter  VI. 
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H.  LITERATURE  REVIEW 


A.  HISTORICAL  PREDECESSORS  AND  THE  INTERNET 

The  U.S.  President  Dwight  Eisenhower  established  the  Advanced  Research 

Project  Agency  (ARP A)  in  1958  as  part  of  the  Defense  Department  in  order  to  improve 
the  new  national  commitment  to  science  and  to  bolster  national  defense  (Moschovitis  and 
others,  1999,  pp.  35-36).  ARPA  began  to  explore  the  possibilities  of  information 
processing  and  computers  under  the  visionary  leadership  of  Joseph  C.  D.  Licklider. 
Searching  for  new  ways  to  improve  human  productivity,  ARPA  conducted  extensive 
research  on  the  feasibility  of  connecting  computers  across  long  distances.  In  1965,  Larry 
Roberts  connected  his  computer  in  the  Boston  area  via  a  phone  line  with  another  in 
California,  which  was  the  entry  into  an  entirely  new  era:  the  era  of  the  Internet. 

Licklider  and  his  team  imagined  a  universal  computer  network.  Ted  Nelson 
coined  the  terms  ‘hypertext’  and  ‘hyperlink’  in  1965,  and  in  1969  ARPAnet  is  bom:  the 
first  multiple-site  computer  network,  sponsored  by  the  U.S.  Department  of  Defense. 

ARPAnet  started  as  an  experimental,  four-computer  network.  By  1971,  ARPAnet 

linked  almost  two  dozens  sites,  including  MIT  and  Harvard.  By  1974,  there  were  over 

200  sites.  During  the  1980s,  more  and  more  computers  using  different  operating  systems 

were  connected.  In  1983,  the  military  portion  of  ARPAnet  was  moved  onto  MILnet,  and 

ARPAnet  was  officially  disbanded  in  1990.  In  the  late  1980s,  the  National  Science 

Foundation’s  NSFnet  began  its  own  network  and  allowed  everyone  to  access  it  It  was, 

however,  primarily  the  domain  of  ‘techies’,  computer-science  graduates,  and  university 

professors.  Finally,  in  1991  Tim  Berners-Lee  developed  the  World  Wide  Web  at  CERN 

(Centre  European  pour  la  Recherche  Nucleaire  -  European  Laboratory  for  Particle 
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Physics)  in  Geneva.  This  event  definitely  opened  the  door  to  the  unlimited  development 
of  the  Internet  society. 

According  to  the  newest  surveys  (Nielsen/NetRatings,  online  resource,  December 
2000),  today  over  153  million  Americans  use  the  Internet  from  their  homes.  Accesses  to 
the  Internet  increases  by  5.3%  monthly,  and  trends  are  similar  in  many  other  countries 
such  as  the  United  Kingdom,  Japan,  and  Scandinavian  countries.  The  process  of  gradual 
integration  of  telephony,  radio  broadcasting,  and  TV  with  the  Internet  is  rapidly 
increasing.  It  is  predicted  that  by  2005  over  one  billion  people  may  be  connected  to  the 
Internet  (Simon  &  Schuster  New  Millennium  Encyclopedia  and  Home  Reference 
Library,  2000,  topic  ‘What  is  the  Internet’). 

The  speed  at  which  the  Internet  is  spreading  is  incredible.  Radio  existed  38  years 
before  50  million  people  tuned  in.  It  took  television  13  years  to  reach  50  million  viewers. 
After  the  introduction  of  the  first  personal  computer  kit,  it  took  16  years  before  50  million 
people  were  using  one.  However,  within  only  four  years  of  the  Internet  being  available  to 
the  general  public,  50  million  people  were  connected  (Simon  &  Schuster,  same  topic). 

B.  EVALUATION  OF  DL  EFFECTIVENESS 

Many  practitioners  have  recognized  evaluation  as  an  integral  part  of  a  lifecycle 

methodology  in  any  instructional  systems  design  (ISD)  model.  Course  evaluation  is  a 

consistent  part  of  the  ‘UCF  model’  of  DL  practice  also. 

Belanger  and  Jordan  (2000)  describe  the  following  model,  called  ‘ADDIE’,  as 
most  commonly  used.  It  includes  five  key  phases: 
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Develop  instructional  objectives  and 
strategies.  Establish  instructional  content 
and  create  storyboards. 

Identify  requirements,  resources,  issues, 
and  constrains. 

Program  interfaces,  databases, 
communication  facilities  for  text,  audio, 
video.  Test  and  document.  Develop  support 
procedures. 

Delivering  of  the  course. 


Evaluate  effectiveness  of  course  and 
delivery  mechanisms. 


Figure  2.  ADDIE  Model  for  DL  Projects  [From:  Belanger  and  Jordan  (2000),  p.  90]. 

Even  a  quick  examination  of  relevant  literature  shows  that  there  is  a  large  amount 
of  research  on  the  effectiveness  of  web-based  or  web-enhanced  education  and  training. 
However,  few  researchers  have  tried  to  capture  several  aspects  of  the  process 
simultaneously.  An  education  or  training  process  is  a  complex,  multidimensional  activity. 
Concentrating  on  only  some  aspects  of  the  process  may  miss  important,  subtle  factors 
important  for  overall  effectiveness. 

There  are  not  many  theoretical  models  analyzing  training  effectiveness.  The  most 
useful  are  Kirkpatrick’s  Four  Levels  of  Evaluation  model,  Marshall  and  Shriver’s  Five 
Levels  of  Evaluation  Model,  and  Van  Slyke’s  Framework.  This  thesis  anchors  these 
training  effectiveness  models,  using  SS301 1  as  a  data  source,  to  determine  which  of  these 
models  is  effective  in  assessing  web-based  instruction. 

Kraiger,  Ford  and  Salas  (1993)  offered  the  following  classification  of  learning 
outcomes: 
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LEARNING 


-  Figure  3.  Kraiger,  Ford  and  Salas’  Classification  of  Learning  Outcomes.  [From: 

Kraiger,  Ford  &  Salas  (1 993),  pp.  3 1 1-328]. 

Their  model  systematizes  learning  outcomes  into  three  basic  clusters  and  analyzes 
each  of  them  in  depth.  Cognitive  outcomes  in  their  model  are  very  close  to  Kirkpatrick’s 
knowledge  transfer,  affective  outcomes  close  to  learner  reactions,  and  skill-based 
outcomes  to  behavior  transfer. 


Kirkpatrick’s  model  [Alliger  and  others  (1997),  pp.  342-343;  Belanger  and  others 
(2000),  pp.  191-192]  is  still  the  most  influential  and  is  a  starting  point  for  other  models.  It 
differentiates  between  four  categories  of  training  criteria:  Learner  reactions;  learning  or 
knowledge  transfer;  behavior  transfer;  and  organizational  impact.  Researchers  often  use 
this  model  to  evaluate  the  impact  of  the  learning  experience  at  affective,  cognitive, 
behavioral,  and  organizational  levels. 
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According  to  Kirkpatrick’s  model,  learner  reaction  as  a  first  level  of  evaluation  is 
affective  in  nature,  which  can  be  assessed  by  surveys,  questionnaires,  focus  groups,  and 
interviews.  What  is  essential  is  that  this  level  is  a  personal  reaction  to  the  learning 
experience.  Knowledge  transfer  is  about  what  is  actually  learned.  It  is  hard  to  measure 
this  level  of  learning,  but  it  is  most  essential  in  measuring  the  effectiveness  of  the 
learning  experience,  compared  to  the  other  learning  objectives.  Knowledge  tests  are  often 
present  in  DL  courses,  which  enables  the  learner  to  assess  his  own  progress  and  obtain 
feedback.  Another  component  in  Kirkpatrick’s  model,  behavior  transfer  assessment,  deals 
with  measurable,  observable  behavioral  changes.  Unfortunately,  this  component  is  often 
not  measured.  Assessment  methods  at  this  level  of  evaluation  include  time-consuming 
360  degrees  surveys.  Organizational  impact  measures  are  usually  focused  on 
improvements  (service  response  time,  service  quality,  etc.)  that  increase  the  return  on 
investment.  These  are  not  causal  measures  and  are  outside  the  scope  of  this  thesis. 

Baldwin  and  Ford  (1988)  describe  training  design  as  one  of  the  main  training 
input  factors  in  the  following  theoretical  model  of  the  knowledge  transfer  process  (Figure 
4).  la  terms  of  their  model,  this  thesis  focuses  on  learner  characteristics  and  learning 
design  as  inputs,  and  learning  and  retention  as  outputs.  Furthermore,  relationships 
between  them  are  of  special  importance  for  this  research  because  the  system  approach  by 
definition  focuses  on  relationships  between  elements  and  subsystems  inside  the  system, 
as  well  as  input  into  and  output  from  the  system. 
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Learning  Inputs  Learning  Outputs  Conditions  of  Transfer 


Figure  4.  Baldwin  and  Ford’s  Model  of  Learning  Design  [From:  Baldwin  &  Ford 

(1988),  p.  65]. 


Another  very  interesting,  and  for  this  thesis  a  useful  model,  is  proposed  by  Van 
Slyke,  Kittner  and  Belanger  (1998).  It  suggests  that  multiple  variables  must  be  taken  into 
account  concurrently  in  looking  at  their  effect  on  distance  education  [Belanger  &  Jordan 
(2000),  pp.  187-190].  Their  framework  targets  distance  education,  not  distributed 
learning  in  the  sense  I  defined  and  used  in  this  thesis.  Their  theoretical  model  addresses 
the  wider  spectrum  of  learning  environments,  including  VTT,  WBT,  and  others.  They 
emphasize  that  their  framework  can  be  used  to  conceptualize  the  learning  environment 
where  the  learner  is  embedded  in  all  its  complexity.  All  key  dimensions  are  present  in  the 
model:  the  institution,  the  learner,  the  course,  and  the  type  or  types  of  distance  education 
applied.  Moreover,  they  stress  that  all  evaluation  programs  should  explicitly  define  which 
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of  these  categories  of  variables  are  being  measured,  which  are  being  controlled,  and  what 
the  outcome  variables  are. 

Belanger  and  Jordan  go  a  step  further,  adapting  Van  Slyke’s  model  (Figure  5). 


Learner 

Institution 

Course 

DL 

Characteristics 

Characteristics 

Characteristics 

Characteristics 

FIT 


Learner 

Instructor 

Society 

Institution 

Figure  5.  Framework  for  Distance  Education  and  Training  [From:  Belanger  & 

Jordan  (2000),  p.  189]. 

This  framework,  which  builds  on  the  previously  mentioned  models,  gradually 
evolved  from  the  analysis  of  learning  outcomes  to  a  very  broad  view  including  even 
society.  Comparing  Baldwin  &  Ford’s  model  with  Belanger  and  Jordan’s,  we  find  that 
new  determinants  and  categories  of  outcomes  are  added.  Learner  characteristics  are 
present  in  both  models.  While  Baldwin  and  Ford  outline  learning  design,  Belanger  and 
Jordan  differentiate  course  characteristics  and  DL  characteristics.  The  last  learning  input 
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for  Baldwin  and  Ford  is  the  work  environment,  but  for  the  latter  two  authors  it  is 
institution  characteristics  and  society.  Belanger  and  Jordan  (2000,  p.  186)  comment: 

All  of  these  characteristics  and  variables  must  be  carefully  examined,  not 
in  isolation,  but  together  in  evaluating  the  appropriateness  for  an 
individual,  a  course,  and  an  organization  to  attend,  convert,  or  implement 
DL.  We  want  to  extend  the  model  to  include  four  different  units  of 
analysis  impacted  by  DL:  the  learner,  the  instructor,  the  institution,  and  the 
society. 

They  also  add  that  there  are  two  levels  of  outcomes  and  thus  two  measures  of 
success  (or  efficiency):  one  for  institutions  and  one  for  students.  As  Figure  5  shows,  they 
extended  the  number  of  units  of  analysis  impacted  by  DL  to  four:  the  learner,  the 
instructor,  the  institution,  and  society. 

C.  OTHER  FACTORS  OF  DL  EVALUATION 

Phipps  and  Merisotis  (1999),  after  analyzing  a  large  amount  of  original  studies  on 

DL  since  1990,  concluded  that  most  of  the  studies  measured  effectiveness  of  DL  in  one, 
less  frequently  two,  and  very  rarely  in  three  of  the  following  areas: 

•  Student  outcomes,  such  as  grades  and  test  scores 

•  Student  attitudes  about  learning  through  distance  education. 

•  Overall  student  satisfaction  toward  distance  learning 

In  other  words,  most  of  the  studies  concentrated  only  on  some  aspects  of  the 

learning  process  and  neglected  other  relevant  parts  of  the  system.  They  found  serious 
methodological  weaknesses  in  most  of  the  studies  and  concluded  that  there  was  a  paucity 
of  original,  legitimate  research  dedicated  to  explaining  or  predicting  phenomena  related 
to  DL.  According  to  their  review,  approximately  28%  of  DL  research  studied  two-way 
interactive  video  technologies,  28%  studied  computer-mediated  learning,  16%  studied 
one-way  broadcast  technologies,  J3%  one-way  prerecorded  video,  6%  two-way  audio 
and  one-way  video  learning,  and  9%  studied  other  technologies.  The  main  research 
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methods  applied  were:  experimental  51%,  descriptive  31%,  case  study  15%,  and 
correlational  3%. 

Suchan  and  Crawford  (1998)  analyzed  implementation  of  network-based 
instruction  (NBT)  to  senior  Navy  medical  executives,  emphasizing  in  introduction  the 
unique  effects  of  technology  on  learner  and  on  interaction  between  learner  and  instructor, 
as  well  as  between  learners.  Their  thorough  analysis  provided  several  recommendations 
at  different  organizational  levels  of  a  learning  process.  Here  is  the  selection  of 
recommendations  most  relevant  to  a  course  like  SS301 1 : 

•  Formal  organizational  arrangements:  provide  appropriate  technology 
and  training  to  use  that  technology;  provide  technological  support. 

•  Informal  organization:  program  evaluation  should  be  conducted  at 
several  points  in  time  to  account  for  the  effects  of  exposure  to  the 
medium;  use  a  qualitative  methodology,  i.e.,  interviews,  to  capture  factors 
that  might  be  not  anticipated  by  instructional  designers. 

•  Instructional  task  design  factors  -  perceived  meaningfulness:  show 
clear  indication  of  the  module’s  purpose;  define  the  module  learning  states 
from  a  personal  and  organizational  perspective;  ensure  that  instructional 
objectives  are  reflected  in  all  evaluation  measures;  develop  an  overview  of 
topics  to  be  covered  to  further  provide  learning  constraints;  implement 
constant  summaries  that  rearticulate  module  purpose. 

•  Instructional  task  design  factors  —  perceptions  of  choice:  create  hot 
links  (hypermedia)  to  other  Web  sites;  monitor  and  feedback  learners’ 
choice  of  information  search  strategies  so  that  they  become  more  aware  of 
the  search  patterns  and  structures  they  use;  use  open-ended  questions 
based  on  readings,  scenarios;  encourage  (students)  to  share  their  own  and 
respond  to  other’s  experiences  in  these  electronic  venues;  provide 
immediate  and  direct  access  to  a  ‘real’  instructor  for  problems  and 
feedback  on  the  program. 

•  Instructional  task  design  factors  -  perception  of  competence  and 
progress:  provide  carefully  timed,  ongoing  feedback  on  progress  toward 
achieving  module  goals;  vary  feedback  style  based  on  module  learning 
outcomes;  use  positive  language  to  frame  all  types  of  electronic  feedback. 


19 


The  authors  conclude  their  recommendations  with  very  important  point  that 
feedback  is,  in  essence,  interactivity,  what  is  of  exceptional  relevance  for  this  thesis. 

Moskal  and  Dziuban  (2000)  in  introduction  of  their  paper  ‘Present  and  Future 
Directions  for  Assessing  Cyber-Education’  analyzed  evaluation  strategies  from 
standpoint  of  the  University  administration.  According  to  them,  evaluation  is  important  to 
determine  both  the  initiative’s  progression  and  ultimately  its  success.  It  is  important,  they 
explain,  to  track  longitudinally  the  impact  of  on-line  courses  on  students  and  faculty.  In 
summary,  they  outline  five  assumptions  of  web-based  teaching  and  learning  evaluations: 

•  Research  should  inform  practice 

•  Evaluation  must  be  contextually  relevant 

•  Faculty  must  be  rewarded  as  research  leaders 

•  Evaluation  must  be  seen  as  an  integral  part  of  the  initiative 

•  As  evaluation  matures  its  demonstrable  impacts  must  be  obvious. 

The  authors  analyzed  in  detail  data  on  both  students  and  the  faculty  at  UCF. 

Eighty-nine  percent  of  web  course  students  were  very  favorable  about  taking  another 

course,  and  70%  of  them  indicated  they  would  definitely  take  another  on-line  course.  As 

main  reason  for  enrolling  on-line  course,  students  mostly  indicated  convenience  of  use. 

The  faculty  teaching  web  courses  had  an  increased  workload,  but  80.4%  of  them  were 

satisfied  or  very  satisfied,  and  93.6%  of  them  would  probably  or  definitely  teach  another 

web-based  course.  They  reported  that  the  main  negative  on-line  aspects  were  time 

demands  (44%),  technical  problems  (38%),  and  student  problems,  mostly  with  accessing 

the  course  content  or  communicating  with  the  instructor.  The  authors  concluded  that 

‘measuring  the  success  of  on-line  teaching  and  learning  is  complex  procedure  nested  in 

series  of  intricate  interactions’  (p.31),  and  that  ‘outcome  comparisons  associated  with 
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web-based  learning  and  traditional  on-campus  education  may  be  intuitively  appealing, 
but  they  do  not  facilitate  responsive  programs.’ 

Hartman,  Dziuban,  and  Moskal  (1999)  in  their  conclusion  inducted  that  ‘faculty 
satisfaction  and  student  outcomes  covary  when  predicting  success  in  ALN 
(Asynchronous  Learning  Network)  programs’.  This  has  implications  for  this  thesis 
because  SS3011  course  was  taught  in  learning  environment  very  close  to  ALN  as  the 
authors  of  the  article  defined  it,  and  because  the  instructor  of  the  course  also  represented 
important  source  of  data  in  this  thesis  research. 

Alliger,  Tannenbaum,  Bennet,  Traver,  and  Shotland  (1997)  conducted  meta¬ 
analysis  of  34  studies  and  found  utility-type  reaction  measures  be  more  strongly  related 
to  learning  than  affective-type  reaction  measures.  This  finding  has  direct  implications  for 
evaluation  of  training  effectiveness. 

Another  important  paper  is  an  article  by  Tannenbaum,  Mathieu,  Salas,  and 
Cannon-Bowers  (1991),  where  authors  found  support  for  their  hypotheses  that  training 
fulfillment,  trainee  reactions,  and  training  performance  are  related  to  post-training 
organizational  commitment,  academic  self-efficacy,  physical  self-efficacy,  and  training 
motivation.  The  authors  provide  (p.  759)  Bandura’s  definition  of  self-efficacy  as  ‘the 
belief  in  one’s  capability  to  perform  a  specific  task’.  The  authors  added  that  self-efficacy 
is  also  related  to  an  individual’s  openness  to  experiment  and  to  the  likelihood  of  using 
new  technology.  Another  factor  discussed  in  their  article  was  organizational  commitment. 
They  defined  it  as  the  relative  strength  of  an  individual’s  identification  with  and 
involvement  in  a  particular  organization. 
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Hill,  Smith,  and  Mann  (1987)  investigated  the  relation  between  sense  of  efficacy 
regarding  computers  and  people’s  readiness  to  use  them.  They  showed  expected  relation 
between  efficacy  beliefs  with  respect  to  computers  and  the  likelihood  of  using  computers. 
Also,  they  found  previous  experience  with  computers  is  related  to  beliefs  of  efficacy  with 
respect  to  computers.  However,  the  latter  did  not  exert  a  direct  independent  influence  on 
the  decision  to  use  computers.  They  concluded  that  their  studies  demonstrated  the 
importance  of  efficacy  in  the  decision  to  adopt  an  innovation  -  it  is  an  important  factor  in 
determining  an  individual’s  decision  to  use  an  innovation.  Experience  per  se,  they  say  (p. 
313),  does  not  directly  affect  subsequent  behavior  regarding  further  adoption  of  computer 
technology.  Rather,  experience  with  computer  technology  leads  to  a  higher  likelihood  of 
technology  adoption  only  through  changes  in  perceived  efficacy.  However,  because  the 
students  in  this  sample  are  officers  of  U.S.  services  in  contemporary  circumstances,  they 
do  not  have  much  choice  about  using  or  not  using  computers;  it  is  everyday  need. 


D.  STUDENT  BEHAVIOR  PATTERNS  ON-LINE 

William  A.  Long's  theory  of  reactive  behavior  is  described  in  several  articles: 

Long  (1985),  Dziuban  and  Dziuban  (1997),  and  Dziuban,  Moskal,  and  Dziuban  (1999). 

In  brief,  it  is  a  theory  based  on  normal  behavior  patterns  that  adolescents  exhibit 
in  a  learning  environment,  explaining  how  young  people  react  to  their  environment  and 
its  associated  stresses.  According  to  Long,  individuals  tend  to  one  of  four  behavior  types, 
combined  with  zero  to  four  ancillary  traits.  Four  types  of  reactive  behavior  patterns  are: 

•  Aggressive  Independent  (AI).  These  students  tend:  to  have  high  energy 
level  and  little  need  for  peer  or  teacher/instructor  approval;  to  lack 
judgment,  express  their  thoughts  and  feelings  impulsively;  to  be 
disorganized  and  nonlinear,  preferring  to  work  independently,  often  in 
leadership  positions.  They  are  challenging  students,  preventing 
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instructor’s  complacency.  Confrontation  resolves  their  ambiguity,  stress, 
or  indecision. 

•  Aggressive  Dependent  (AD).  These  students  are  high  achievers,  usually 
very  active  in  student  organizations,  honor  courses,  etc.  They  are  non- 
confrontational,  eager  to  please,  participate  in  class. 

•  Passive  Independent  (PI).  These  students  can  be  non-communicative, 
non-participatory,  and  stubborn.  Even  if  having  superior  abilities,  they 
may  behave  contrary  to  their  own  best  interests. 

•  Passive  Dependent  (PD).  They  are  affectionate,  sensitive,  non- 
confrontational,  and  extremely  compliant.  Their  need  for  approval  is  high, 
dominates  their  relations  with  peers  and  with  instructor.  They  see 
disagreements  and  criticism  as  personal  rejection.  When  mature,  they  can 
become  very  caring  and  gentle  persons. 


According  to  Long’s  theory  and  many  empirical  findings,  any  of  these  four 
reactive  behavior  patterns  is  associated  with  none  or  any  combination  of  next  four 
ancillary  traits  (descriptions  derived  from  Dziuban,  Moskal,  and  Dziuban  [1999]): 

•  Phobic  Trait.  These  students  tend  to  develop  focused  fears  concerning 
possible  undesired  outcomes  of  their  academic  learning  process;  they  are 
cautions  decision  makers,  but  excellent  students  in  analytical  situations. 

•  Impulsive  Trait.  They  are  erratic  and  capricious,  adding  vitality  and 
energy  to  the  classroom.  They  tend  to  answer  a  question  before  it  is 
completely  asked,  or  to  undertake  an  on-line  course  before  technical 
requirements  are  met. 

•  Obsessive-Compulsive  Trait  These  students  are  thorough,  methodical, 
and  careful,  very  organized.  They  strongly  tend  to  succeed  in  completing 
their  tasks  and  they  enjoy  academic  achievement  and  excellence,  but  may 
lack  spontaneity. 

•  Hysterical  Trait  These  students  are  creative,  helpful,  compassionate,  and 
dramatic.  They  may  create  contagious  energy  and  enthusiasm  in  classroom 
environment,  but  in  crises  they  become  chaotic  and  may  fail  in  their 
assignments. 


Long  and  other  researchers  confirmed  the  validity  of  this  theory  in  several  studies, 
including  studies  of  congruence  of  parents  and  teachers,  studies  of  predicting  types  and 
traits  of  students  from  teachers’  perception  of  their  academic  problems,  studies  of  the 
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correspondence  of  phenomenological  and  self-report  approaches,  etc.  All  these  types  and 
traits  are  present  in  normal  population  of  students,  so  the  theory  has  instructional  utility 
in  higher  education. 

Dziuban,  Moskal,  and  Dziuban  (1999)  found  indications  that  students  in  on-line 
courses  tended  not  to  be  independent  learners.  Furthermore,  passive  dependent  students, 
who  are  usually  rather  observers  than  participants,  reacted  least  favorably  to  on-line 
courses.  The  authors  also  suggest  directions  for  further  research  and  conclude  that  lack  of 
understanding  and  evaluation  of  described  types  of  reactive  behavior  patterns  may 
unknowingly  discourage  significant  segments  of  student  population  from  enrolling  in 
distributed  learning  programs.  Commenting  on  this  finding,  Dziuban  and  others  (2000) 
concluded:  ‘The  virtual  environment  is  one  in  which  students  must  reevaluate  their 
fundamental  approach  to  learning’. 
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m.  METHODOLOGY 


The  organization  of  this  chapter  is  as  follows: 

•  Case  Analysis 

•  Demographic  Data  &  Biographies 

•  Questionnaires  On-Line 

•  Student  Tracking,  Student  Profiles,  and  Pages  Tracking 

•  Grades 

•  Student  Opinion  Forms  (SOFs) 

•  Interviews  with  the  Students 

•  Interviews  with  the  Instructor 

The  initial  idea  for  this  thesis  was  triggered  during  preparations  for  the  first  on¬ 
line  delivery  of  the  course  ‘Space  Systems  -  Technology  and  Applications’  (course  code 
SS301 1),  taught  during  summer  quarter  of  academic  year  2000  at  the  Naval  Postgraduate 
School  (NPS)  in  Monterey,  California.  Commander  Sue  Higgins,  US  Navy,  the  instructor 
of  this  course,  was  previously  trained  at  the  University  of  Central  Florida  (UCF), 
Orlando,  Florida,  so  the  resulting  teaching/leaming  during  the  course  process  could  be 
interpreted  as  an  application  of  ‘UCF-model  of  distributed  learning’  to  one  course  at 
NPS.  Furthermore,  UCF  provided  continuous  support  during  the  preparations  for  and 
during  the  course  in  hardware  (servers),  software  (maintaining  the  web  pages  and  forums, 
removing  ‘bugs’,  and  necessary  technical  troubleshooting  and  help),  and  in  other  ways 
(occasional  meetings,  consultations,  etc.). 

A.  CASE  ANALYSIS 

Case  study  by  definition  means  a  conceptual  umbrella  for  multiple  sub  studies  of 

an  isolated  example  (Yin,  1993,  p.  xii),  which  is  to  say  that  different  sources  of  data  are 
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gathered.  Ultimately,  this  thesis  focuses  on  describing  and  analyzing  qualitative  data,  but 

the  organization  of  the  course  and  of  this  research  enabled  the  collection  of  significant 

amount  of  quantitative  data.  This  case  study  is  composed  of  several  sub-studies  based  on: 

o  Basics  on-line  questionnaire 

o  A-2  questionnaire 

o  Module  14  questionnaire 

o  Final  Survey  questionnaire 

o  On-line  tracking  data 

o  Interviews  with  the  students 

o  Interviews  with  the  instructor 

o  Final  grades 

All  these  sub-studies  are  combined  and  integrated  with  the  primary  and  secondary 
research  questions. 

The  organization  of  the  case  study  was  dictated  by  the  dynamics  of  the  course  and 
methods  applied,  i.e.  the  researcher  had  to  adapt  to  the  teaching  schedule  of  the  course  to 
avoid  any  interference. 

The  four  evaluation  instruments  were  put  on-line  at  appropriate  times,  according 
to  targeted  parts  of  the  course  or  modules  that  I  wanted  to  evaluate. 


The  instructor  and  I  made  clear  the  purpose  of  the  thesis  to  the  students  on  several 
occasions.  The  instructor  introduced  me  to  the  students  during  the  first  class,  so  they 
could  become  familiar  with  me  and  with  the  research  I  planned  to  conduct. 

I  organized  the  interviews  with  the  students  individually  several  weeks  after  the 
course  when  exams  and  final  papers  from  their  other  courses  were  completed.  I  used  that 
time  because  they  were  free  of  other  academic  pressures,  so  they  could  be  more  relaxed, 
objective,  and  willing  to  be  frank  and  vocal  during  the  interviews. 
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Finally,  I  interviewed  the  instructor:  before,  during,  and  after  the  course.  These 
latter  occasions  were  much  more  directed  by  my  observations,  by  questions  that  appeared 
from  other  data,  or  by  her  opinion  of  what  else  was  important  but  still  not  mentioned. 

Some  of  the  instruments  used  in  this  research  called  for  anonymous  responses, 
and  some  were  associated  with  student  identity  (names).  Consequently,  it  was  possible  to 
explore  relations  between  variables  from  different  instruments  (qualitatively  and 
quantitatively),  including  measures  of  association  (for  instance  coefficient  of  rank 
correlation).  Such  identifiable  sources  of  data  in  this  research  were: 

o  The  Basics  -  Demographics  and  Studying  Strategies  (A-l)  on-line 
questionnaire 

o  Module  14  Evaluation 

o  Biographies  (all  except  two  students  posted  them  to  forum,  as  the 
instructor  requested  in  order  to  enable  the  students  to  get  know 
each  other) 

o  Student  Tracking,  Student  Profiles,  and  Pages  Tracking.  WebCT 
software  was  set  to  automatically  track  for  each  student  the  number 
of  visits  to  web  pages,  number  of  read  articles  in  forums,  number 
of  posts  to  forums,  and  additional  data. 

o  Grades 

o  Individual  interviews 

It  was  not  possible  to  associate  other  questionnaires  ( After-orientation 

Questionnaire  (A-2)  and  Final  Survey)  with  student  identity,  so  analysis  of  responses 

may  only  be  made  in  the  distribution  of  the  answers,  of  associations  of  variables  within 
the  instruments,  and  in  qualitative  analysis.  One  of  the  main  reasons  for  this  limitation 
was  the  way  the  WebCT  software  was  set.  As  measures  of  association,  I  used  two 
different  coefficients  of  correlation,  depending  on  whether  a  variable  was  numeric  or 
ordered  categorically: 
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Spearman’s  Rho  (p)  coefficient  of  rank  correlation  is  applicable  with  ordered 
categorical  variables  and  with  a  given  sample  size.  Some  other  indicators  (based  on  x2 
and  some  others)  could  be  used  too.  The  equation  for  calculating  the  Rho  coefficient  is 
shown  in  Equation  1 . 
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Equation  1.  Spearman’s  Rho  Coefficient  of  Rank  Correlation  [From:  Conover  (1999), 

P-315]. 

For  numerical  variables,  Pearson’s  coefficient  of  correlations  was  calculated: 
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Equation  2.  Pearson’s  r  Coefficient  of  Correlation  [From:  Berk  &  Carrey  (2000),  p. 

321]. 

For  both  coefficients  I  provided  respective  p-values.  In  calculating  p-value  for 
each  of  the  coefficients,  the  null-hypothesis  (H©)  is  that  variables  in  the  questions  are 
mutually  independent,  while  alternative  hypothesis  (Ha)  finds  a  tendency  either  for  the 
larger  values  of  Xj  to  be  paired  with  the  larger  values  of  Y*,  or  a  tendency  for  smaller 
values  of  Xj  to  be  paired  with  the  larger  values  of  Yi  in  which  case,  the  coefficient  would 
have  negative  sign.  Due  to  the  small  sample  in  this  research,  a  p-value  of  0.05  would  be 
acceptable  as  statistically  significant. 
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B.  DEMOGRAPHIC  DATA 


Only  selected  demographic  data  were  collected,  mostly  within  the  first  on-line 
questionnaire  (see  next  section).  Not  all  data  from  this  set  is  applicable  to  this  thesis,  so 
part  of  these  data  is  reported  in  Appendix  F.  Distributions  and  other  relations  are 
explored  in  Chapter  IV. 

C.  QUESTIONNAIRES  ON  LINE 

Thirty-two  students  attended  SS3011,  but  only  thirty  of  them  completed  it  as 
regular  NPS  students:  two  of  them  were  not  regular  NPS  students  and  they  were  not 
getting  credit  for  finishing  the  course.  Moreover,  they  were  off  campus  as  non-resident 
additional  attendees  of  the  course  and  were  treated  differently  from  the  other  students. 
For  these  reasons,  they  are  excluded  from  the  sample. 

Four  on-line  questionnaires  were  administered  to  the  students  (Appendices  A 
through  D).  The  instructor’s  support  staff  from  the  Training  System  Division,  Naval  Air 
Warfare  Center,  Orlando,  Florida,  provided  the  initial  set  of  questions,  but  many  items 
were  adapted  and  modified.  Also,  a  significant  number  of  new  questions  were  created 
because  of  the  needs  of  the  instructor  and  the  research  questions  for  this  thesis.  A  new 
instrument  was  incorporated  into  the  last  questionnaire:  Long-Dziuban’s  instrument  with 
slightly  modified  instructions.  The  authors  approved  its  application  in  this  thesis.  The 
four  questionnaires  were: 

•  The  Basics  -  Demographics  and  Studying  Strategies  (A-l).  This 
instrument  covered  general  facts  about  the  students,  but  focused  mostly  on 
what  the  instructor  was  interested  in  order  to  know  her  students  better. 
Besides  questions  about  e-mail  address,  student  status,  service,  pay  grade 
and  MOS,  there  are  also  questions  about  previous  experiences  relevant  for 
attending  the  course  (former  space-related  duties  or  courses  attended, 
experience  in  using  web  resources),  about  expectations  (expected  number 
of  web  visits  weekly,  overall  expectation  on  the  on-line  course),  self- 
perceived  proficiency  in  using  computers  (proficiency  in  using  computers. 
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on-line  research  techniques),  and  concern  about  one’s  own  ability  to  use 
the  web-based  materials  in  the  class.  The  questionnaire  also  provided  the 
students  an  opportunity  to  make  comments  in  their  own  words.  It  was  put 
online  at  the  end  of  first  module.  It  had  a  total  of  25  items. 

•  After-orientation  Questionnaire  (A-2).  This  instrument  contained  26  items. 
It  was  directed  towards  the  students’  comfort  level  on  various  aspects  of 
the  course,  and  was  administered  after  the  first  two  weeks  of  the  course. 
For  instance,  the  students  were  asked  about  setting-up  their  systems, 
accessing  the  Internet,  word  processing,  posting  to  forums,  sending  e-mail, 
using  WebCT  software,  on  technical  support,  and  so  on.  Also,  they  had 
several  items  where  they  could  respond  in  their  own  words  and  comment 
on  different  issues  about  their  first  two  weeks  of  the  course.  Several 
questions  explicitly  asked  them  about  their  comfort  level  with  the  most 
important  course  features,  like  forums,  e-mail,  quizzes,  links,  and  others. 

•  Module  14  Evaluation  was  applied  after  the  fourteenth  module.  Module  14 
was  especially  interesting  for  two  reasons.  First,  it  was  different  from 
other  modules,  and  therefore,  deserved  special  attention.  Second,  that 
point  in  time  was  close  to  the  end  of  the  course,  so  it  was  a  very  good 
moment  to  ask  the  students  overall  questions  about  the  course.  This 
questionnaire  was  the  shortest  (only  14  items),  and  there  were  three  main 
groups  of  questions: 

o  Questions  directly  related  to  previously  specified  learning 
objectives  (about  abilities  to  identify  key  elements  in  Space 
Control,  abilities  to  synthesize  others’  findings  in  forums, 
readiness  to  interact  with  subject  matter  expert  in  the  area,  etc.), 
o  Questions  about  final  comfort  level  in  accessing  the  Internet, 
sending  e-mail,  and  posting  comments  to  forums, 
o  Open-ended  question  to  provide  comments  and  additional 
thoughts. 

•  The  Final  Survey  was  applied  after  the  final  exam/paper,  but  before  the 
students  received  their  final  grades  for  the  course.  It  had  34  questions  and 
contained  the  following  groups  of  items: 

o  Questions  the  instructor  found  very  interested,  especially  in  the 
context  of  the  ultimate  learning  objectives  of  the  course, 
o  Questions  about  the  perceived  useftdness  of  several  features  used 
during  the  course  (guidelines  at  the  course  homepage,  quizzes, 
module  readings,  textbook  readings,  integration  of  responses  to 
forums). 

o  Questions  about  affective  reactions  (liked/disliked)  to  several 
features. 

o  Open-ended  questions  about  the  instructional  strategies  and 
materials,  usability  of  everything  they  learned  in  the  course, 
flexibility,  fundamental  changes  in  the  approach  to  learning,  and 
similar  topics. 
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o 


Four  questions  about  the  interactivity  as  an  important  aspect  of  the 
learning  process, 
o  Long-Dziuban’s  on-line  reactive  patterns  instrument 

Though  it  may  seem  that  four  on-line  questionnaires  during  the  course  took  too 
much  of  the  students’  time,  data  about  the  time  the  students  actually  spent  in  answering 
the  questionnaires  clearly  show  that  it  was  not  the  case.  At  most  they  needed  between 
three  and  five  minutes  per  questionnaire,  and  the  majority  of  them  chose  to  answer  all  or 
almost  all  of  the  questionnaires. 

Thirty  of  the  students  answered  the  first  questionnaire;  thirty-one  the  second  one; 
twenty-six  the  third  questionnaire;  twenty-one  answered  the  Final  Survey  questionnaire. 


D.  STUDENT  TRACKING,  STUDENT  PROFILES,  AND  PAGES  TRACKING 
WebCT  software  had  installed  a  system  of  tracking  student  on-line  activities: 

o  Total  number  of  visits  to  the  course  pages  (also  called  total  number 
of  hits) 

o  Total  number  of  visits  to  the  course  homepage 

o  Total  number  of  visits  to  the  course  content  page 

o  Total  number  of  forum  articles  (posts)  read 

0  Total  number  of  forum  articles  (posts)  posted 

o  Total  number  of  original  forum  articles  posted  (i.e.  not  as 
responses  to  someone’s  other  posts) 

o  Total  number  of  follow-up  forum  articles  posted  (i.e.  as  responses 
to  someone’s  other  posts) 

o  Date  and  time  of  posting  (for  each  forum  post) 

o  Number  of  different  course  pages  visited 

E.  STUDENT  OPINION  FORM  (SOFS) 

Student  Opinion  Forms  are  the  standardized  and  mandatory  formal  way  of 

obtaining  feedback  for  any  course  taught  at  NPS.  They  are  being  applied  not  only  in 
order  to  evaluate  student  perception  of  instructor  and  course  effectiveness,  but  also  as  an 
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additional  help  and  support  to  the  instructor’s  efforts  in  improving  the  course,  the 
methods  of  instruction,  and  other  important  aspects  of  teaching. 

SOFs  are  always  applied  anonymously  on  forms  that  an  equipped  computerized 
system  can  read.  The  current  version  of  SOF  forms  in  use  is  NPS  5040/2  (REV  3-84)]. 
Besides  elementary  data  about  the  course,  on  the  student’s  curriculum,  the  number  of 
hours  during  the  quarter  the  student  attended,  and  the  number  of  quarters  already 
completed,  there  were  16  questions  about  the  course,  the  quality  of  teaching,  the  exams, 
the  textbook,  and  similar  topics.  Students  answer  all  16  questions  on  a  scale  from  5  to  0 
(from  ‘strongly  agree’  to  ‘strongly  disagree,  including  a  category  for  ‘no  comment’).  It 
was  easy  to  quantitatively  analyze  student  feedback,  but  an  appropriate  qualitative 
analysis  of  student  responses  can  also  lead  to  qualitative  conclusions  about  the  overall 
success  of  the  course. 

F.  GRADES 

The  instructor  provided  a  list  of  final  grades  for  all  SS3011  students  who 
successfully  completed  the  course.  The  analyses  include  these  data. 

G.  INTERVIEWS  WITH  THE  STUDENTS 

In  the  beginning  of  the  course,  the  instructor  introduced  me  to  the  students  and 

explained  the  goals  of  this  research.  Near  the  end  of  the  course,  the  instructor  sent  an  e- 
mail  to  the  students  announcing  I  would  like  to  conduct  brief,  post-course  individual 
interviews  with  the  students  who  voluntarily  chose  to  participate.  Then,  I  sent  individual 
e-mail  messages  to  each  of  them. 

I  prepared  eleven  open-ended  questions  (Appendix  E).  As  a  clinical  psychologist 

with  experience  in  hundreds  of  individual  interviews,  questionnaires,  and  other  research 

methods  with  military  officers  and  enlisted  personnel,  I  planned  to  keep  the  interviews 
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focused  on  questions  and  to  intervene  minimally,  only  when  it  was  necessary  to  redirect 
their  comments  and  explanations  back  to  the  questions.  I  intended  to  encourage  the 
students  to  talk  frankly  and  to  feel  free  to  give  critiques,  recommendations,  and 
suggestions  concerning  the  on-line  course. 

The  most  interesting  and  intriguing  responses  and  comments  from  the  four  on-line 
questionnaires,  and  dilemmas  I  expected  in  the  discussion  of  the  research  questions 
primarily  inspired  the  choice  of  these  interview  questions. 

I  decided  to  conduct  the  interviews  one  to  two  months  after  the  course  was 
completed  so  they  would  not  feel  pressure  from  other  final  exams  or  written  assignments 
and  be  more  or  less  free  from  the  ‘fresh’  impressions  of  their  final  grade  in  SS301 1 . 

H.  INTERVIEWS  WITH  THE  INSTRUCTOR 

I  had  over  fifteen  formal  and  informal  meetings  with  the  instructor.  I  listed  areas 

and  prepared  questions,  which  I  then  I  sent  to  her  several  days  before  the  arranged 
interviews,  for  the  purpose  of  direct  collection  of  facts,  her  opinions,  interpretations,  and 
comments  on  relevant  course  issues.  The  interviews  were  conducted  without  a  third  party 
present  and  in  a  pleasant  office  atmosphere.  I  took  notes  as  the  instructor  answered  the 
questions  and  added  comments. 
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IV.  RESULTS  AND  DATA  ANALYSIS 


In  this  chapter  I  provide  data  I  collected  using  all  instruments  in  this  research: 
four  on-line  questionnaires,  Student  Opinion  Forms,  student  on-line  tracking,  final 
grades,  and  interviews  with  the  students  and  with  the  instructor.  Besides  that,  I  made  the 
appropriate  data  analysis,  mostly  concentrating  on  three  key  points:  distributions  of  the 
responses/data,  degree  of  association  between  different  variables  (in  terms  of  coefficients 
of  correlation  or  rank  correlation),  and  qualitative  analysis  of  responses  to  open-ended 
questions  or  interview  questions.  Data  not  relevant  to  research  questions  are  omitted  or 
moved  to  Appendix  F.  After  each  section,  I  provide  a  summary  of  the  data  analysis  in  the 
respective  section  and  a  brief  discussion.  Integration  of  all  results  and  overall  discussion 
appears  in  Chapter  V. 

A.  QUESTIONNAIRES  ON-LINE 

1.  First  questionnaire  -  ‘THE  BASICS,  Your  Demographics  and 
Studying  Strategies’ 

The  first  question  (Appendix  A)  was  about  student  e-mail  addresses,  and  secondly 
about  their  status  as  a  NPS  student.  All  students  were  full-time  NPS  students.  The 
number  of  valid  responses  was  thirty. 

Table  1  shows  students  by  service. 


Service 

Number  of  students 

Percent  (%) 

US  Marine  Corps 

13 

43.34 

US  Navy 

10 

33.33 

4 

13.33 

US  Air  Force 

0 

0.00 

Others 

3 

10.00 

Total 

30 

100.00 

Table  1 .  Students  in  the  Sample  by  Service. 
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Thirteen  officers  (43.33%)  were  Marine  Corps  officers,  one  third  were  Navy 
officers,  and  three  were  US  Army  officers  .  There  were  no  Air  Force  officers,  and  the 
service  of  the  others  was  not  specified. 


Table  2  below  shows  the  distribution  of  the  students  by  pay  grade  (not  rank). 


Pay  Grade 

Number  of  Students 

Percent  (%) 

0-3 

18 

60.00 

0-4 

10 

33.33 

0-5 

1 

3.33 

Not  available 

1 

3.33 

Total 

30 

100.00 

Table  2.  Students  in  the  Sample  by  Pay  Grade. 


Since  pay  grade  is  an  ordered  categorical  category  (except  where  datum  on  pay 
grade  was  not  available),  it  is  possible  to  check  its  possible  associations  with  other 
numerical  or  ordered  categorical  variables  -  not  only  with  such  variables  in  the  first 
questionnaire,  but  also  in  the  Module  14  questionnaire  (which  was  also  identifiable  by 
student’s  name)  and  in  student  on-line  tracking  variables. 

Statistical  indicators  like  the  mean  or  standard  deviation  would  not  be  appropriate 
here  because  responses  to  these  statements  are  ordered  as  categorical  variables  where 
semantic  distances  between  categories  are  only  artificially  assumed  to  be  equal.  Only 
nonparametric  statistical  methods  are  applicable  because  the  assumption  about  normal 
distribution  does  not  have  any  justification  in  this  situation.  Also,  the  sample  is  very 
small. 

For  pay  grade,  only  one  rank  correlation  coefficient  was  statistically  significant 
(Table  3). 
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Instrument 

Statement 

P 

p  value 

First 

questionnaire 

‘Have  you  experienced  an  on-line  course  (or 
courses)  that  used  a  web  site  to  do/complete 
coursework?’ 

0.404 

0.027 

Table  3 .  Significant  Correlation  and  Level  of  Significance  between  Student  Pay 

Grade  and  Another  Variable  . 

Students  with  a  higher  pay  grade  (i.e.  with  higher  rank)  tended  to  have 
significantly  more  previous  on-line  experience. 


The  eighth  item  the  students  were  asked  was  about  the  number  of  quarters  they 
had  been  enrolled,  including  the  current  quarter.  Their  responses  are  analyzed  in  Table  4. 


Quarters  completed 

Number  of  students 

Percent  (%) 

None 

l2 

3.33 

One 

3 

10.00 

Four 

17 

56.67 

Five 

8 

26.67 

Six 

1 

3.33 

Total 

30 

100.00 

Table  4.  Students  in  the  Sample  by  Number  of  Quarters  Enrolled  (Including 

Current  Quarter). 

Three  student  tracking  variables  had  significant  correlations  with  the  number  of 
quarters  enrolled.  Since  all  these  variables  were  numerical,  I  used  Pearson’s  r  coefficient 
of  correlation.  Table  5  shows  significant  correlations. 


2  This  student  probably  didn’t  understand  the  question,  because  it  asked  for  number  of  quarters 
enrolled  including  current  quarter.  Other  data  showed  that  all  students  in  the  sample  had  status  of  regular 
NPS  student. 
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Instrument 

Variable 

r 

p  value 

Student  tracking 

Number  of  visits  to  the  course  homepage. 

-0.479 

0.007 

Student  tracking 

Number  of  posts  to  the  forums. 

-0.432 

0.017 

Student  tracking 

Number  of  follow-up  posts  to  the  forums. 

-0.388 

0.034 

Table  5.  Significant  Correlations  and  Levels  of  Significance  between  the  Number 
of  Quarters  Enrolled  and  Other  Variables  (Ordered  by  Significance). 

Senior  NPS  students  (those  enrolled  in  their  fourth  to  the  sixth  quarter)  slightly, 
but  statistically  significantly,  tended  to  visit  the  course  homepage  fewer  times,  to  post 
fewer  forum  articles,  and  fewer  follow-up  posts  to  the  forums. 

Asked  about  previous  space  tours/duties,  only  two  students  responded  they  had 
‘some  training’.  One  responded  he  had  several  programs,  and  all  other  students  none. 

The  next  question  was  about  previously  attending  any  other  space-related 
course(s).  Only  one  student  had  attended  one  such  course,  and  all  the  others  none. 


Asked  about  previously  attending  any  on-line  courses,  only  three  students 
responded  they  had  such  experience.  One  of  them  was  actually  attending  an  MBA 
program  on-line. 


The  eighteenth  item  was  about  the  expected  frequency  of  using  the  course  web 
site.  The  responses  are  shown  in  Table  6. 


Expected  frequency 

Number  of  students 

Percent  (%) 

3-5  times  per  week 

23 

76.66 

2  times  a  week 

2 

6.67 

At  least  once  a  week 

2 

6.67 

‘I  don’t  know’ 

3 

10.00 

Total 

30 

100.00 

Table  6.  Expected  Frequency  of  Using  the  Course  Web  Site. 
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This  variable  has  significant  correlation  to  only  one  variable:  the  number  of 
original  postings  made  by  the  student  in  forums  (student  tracking).  Spearman’s  p  is 
0.423,  p  is  0.020.  The  students  with  a  higher  than  expected  frequency  of  accesses  tended 
to  send  more  postings  to  forums  during  the  course. 


The  next  item  the  students  were  asked  was  about  where  they  plan  to  use  access  to 
the  course  the  most.  Table  7  shows  the  responses.  This  is  a  non-ordered  categorical 
variable,  so  rank  correlation  is  not  applicable.  One  half  of  the  students  planned  to  access 
the  course  from  their  homes,  which  indicates  they  intended  to  use  the  convenience  of 
accessing  the  on-line  course  from  a  place  other  than  the  school. 


Location 

Number  of  Students 

Percent  (%) 

BOQ  room  i 

0 

0.00 

A  house 

153 

50.00 

Computer  lab  in  campus 

14 

46.67 

1  . 

3.33 

Shared  area  in  living  quarters 

0 

0.00 

Others 

0 

0.00 

Total 

30 

100.00 

Table  7.  Location  that  Students  Expect  to  Use  the  Most  to  Access  the  Course. 

The  next  question  asked  the  students  about  their  self-perceived  level  of  computer 
proficiency:  ‘On  a  scale  of  1  to  5,  where  1  =  novice  and  5  =  expert,  how  would  you  rate 
your  current  proficiency  in  using  computers’?  Figure  6  shows  the  distribution  of  the 
responses.  Most  of  the  students  (seventeen  or  56.67%)  perceived  themselves  at  a 
proficiency  level  between  ‘medium’  and  ‘expert’.  Only  one  student  chose  a  response 
between  ‘novice’  and  ‘medium’.  This  variable  showed  statistically  significant  correlation 
with  three  other  variables  (Table  8). 


3  One  student  added,  answering  the  following  question  -  ‘possibly  at  school  in  one  of  the  labs’. 
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Figure  6.  Distribution  of  Self-Perceived  Computer  Proficiency. 

N  =  30  Mean  =  3.77  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 


Statement 

P . 

P  value 

First 

questionnaire 

‘On  a  scale  of  1  to  5  where  l=novice  and 
5=expert,  how  would  you  rate  your  on-line  search 
resources  and  techniques?’ 

0.836 

<  0.001 

First 

questionnaire 

‘Please  indicate  the  extent  to  which  you  agree 
with  the  following  statement: 

I  expect  this  online  course  will  be  valuable  and 
improve  my  learning:’  [l=strongly  disagree  ... 
5=strongly  agree] 

-0.498 

0,005 

Number  of  articles  read 

0.425 

0.019 

Table  8.  Significant  Correlations  and  Levels  of  Significance  between  Self- 
Perceived  Computer  Proficiency  and  Other  Variables  (Ordered  by  Significance). 

Students  who  perceived  themselves  as  more  computer  proficient  strongly  tended 
to  perceive  themselves  as  better  in  using  on-line  search  resources  and  techniques.  The 
second  relation  (a  much  weaker  correlation  than  the  first  one)  means  that  students  who 
perceive  themselves  as  more  computer  proficient  expected  to  gain  and  learn  less  from 
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this  on-line  course  than  students  who  perceive  themselves  as  not  very  computer 
proficient.  The  third  relation  shows  that  the  students  who  perceived  themselves  as  more 
computer  proficient  tended  to  read  more  articles  in  the  course  forums.  They  probably 
found  the  forums  easier,  more  comfortable,  less  distractible,  could  pay  more  attention  to 
the  articles,  and/or  tend  to  be  more  active  in  reading  the  forums  articles. 

The  next  question  in  the  first  questionnaire  was  about  self-perceived  on-line 
searching  skills:  ‘On  a  scale  of  1  to  5  where  l=novice  and  5=expert,  how  would  you  rate 
your  on-line  search  resources  and  techniques?’  Figure  7  shows  the  distribution  of  the 
responses.  Fourteen  students  (46.67%)  saw  themselves  as  between  ‘medium’  and 
‘expert’,  nine  students  (30.00%)  as  ‘medium’,  while  only  four  students  (13.33%)  thought 
they  were  between  ‘novice’  and  ‘medium’  in  search  techniques. 

Self-Percieved  Search  Techniques 


Number  of 
students 


Novice  Medium  Expert 

9 


Figure  7.  Distribution  of  Self-Perceived  On-Line  Search  Skills. 

N  =  30  Mean  =  3.63  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

Five  correlations  with  other  variables  were  significant  (Table  9). 
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Instrument 

Statement 

P 

p  value 

First 

questionnaire 

‘On  a  scale  of  1  to  5,  where  1  =  novice  and  5  = 
expert,  how  would  you  rate  your  current 
proficiency  in  using  computers.’ 

0.836 

Student  tracking 

Number  of  different  pages  visited 

-0.404 

0.027 

Module  14 

‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or 
destruction  by  adversaries.’  [1  =  strongly 
disagree  . . .  7  =  strongly  agree] 

0.398 

0.030 

First 

questionnaire 

‘Please  indicate  the  extent  to  which  you  agree 
with  the  following  statement: 

‘I  expect  this  online  course  will  be  valuable 
and  improve  my  learning’  [l=strongly  disagree 
...  5=strongly  agree] 

-0.368 

0.045 

Module  14 

‘Evaluating  the  likelihood  that  various  types  of 
space  countermeasures  might  be  used  across 
the  peace- war  continuum.’  [1  =  strongly 
disagree  . . .  7  =  strongly  agree] 

0.367 

0.046 

Table  9.  Significant  Correlations  and  Levels  of  Significance  between  Self- 
Perceived  On-Line  Search  Skills  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  first  relation,  the  only  high  correlation  to  other  variables,  means  that  students 
who  saw  themselves  as  more  computer  proficient  strongly  tended  to  use  on-line  search 
techniques  better.  The  second  relation  is  also  logical:  students  who  think  of  themselves  as 
skilled  in  on-line  search  techniques  expect  to  gain  less  from  the  course  (in  terms  of 
valuable  improvements  in  their  learning),  while  students  with  less  on-line  experience  and 
skill  expect  to  gain  more  because  of  the  on-line  nature  of  the  course. 

The  relations  with  items  from  the  Module  14  questionnaire  are  hard  to  explain. 
Possibly,  students  with  high  on-line  search  skills  (by  their  own  assessment)  later  in  the 
course  achieved  more  in  the  sense  of  the  course  learning  objective  described  in  the 
respective  question.  It  is  interesting  that  the  students  with  higher  level  search  skills 
tended  to  visit  a  smaller  number  of  course  pages.  A  possible  explanation  would  be  that 
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they  were  able  to  get  to  pages  they  wanted  to  visit  easier,  as  opposed  to  students  less 
skilled  in  search  techniques,  who  had  to  use  the  ‘longer  way’  and  thus  visited  more 
different  pages  in  order  to  get  what  they  wanted  from  the  course  web  site. 

The  next  scale  was  about  the  self-perceived  expectations  of  improvement  that  the 
course  would  provide  in  terms  of  their  learning  techniques:  ‘Please  indicate  the  extent  to 
which  you  agree  with  the  following  statement:  I  expect  this  on-line  course  will  be 
valuable  and  improve  my  learning.’  Figure  8  shows  the  distribution  of  the  responses. 

Over  73%  students  expected  the  on-line  course  to  improve  their  learning 
(‘somewhat’  or  ‘strongly’),  and  only  two  of  them  disagreed  with  that  statement.  This 
variable  had  five  significant  correlations  (Table  10).  Correlations  with  other  variables  in 
this  questionnaire  are  telling  us  that  more  experienced  and  skilled  students  expect  to  gain 
less  from  the  course.  The  only  conclusion  to  make  about  the  other  three  correlations  is 
that  expectations  on  learning  improvement  are  somewhat  associated  with  self-assessed 
achievements  in  meeting  learning  objectives  of  Module  14. 


"I  Expect  On-line  Course  to  Improve  my 
Learning" 


Figure  8.  Distribution  of  Expected  Learning  Improvement. 
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N  =  30 
Minimum  =  1 


Mean  =  3.97 
Maximum  =  5 


Median  -  4 
Mode  =  4 


Instrument 

Statement 

P 

p  value 

First 

questionnaire 

‘How  would  you  rate  your  current  proficiency  in 
using  computers?’ 

-0.498 

0.005 

Module  14 

‘Interacting  with  a  subject-matter  expert  whose 
knowledge  you  can  use  to  solve  space-system 
problems.’ 

0.452 

0.012 

Module  14 

‘Identifying  key  elements  of  Space  Control.’ 

0.405 

0.026 

Module  14 

‘Participating  in  forums  (e.g.  discussing, 
identifying/  consulting  subject  matter  experts),  to 
evaluate  space  control  issues.’ 

0.392 

0.032 

First 

questionnaire 

‘How  would  you  rate  your  on-line  search  resources 
and  techniques?’ 

-0.368 

0.045 

Table  10.  Significant  Correlations  and  Levels  of  Significance  between  Expected 
Learning  Improvement  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  next  item  asked  students  about  their  concerns  about  the  course:  ‘I  am 
concerned  about  my  ability  to  use  the  web-based  materials  in  this  class’.  The  distribution 
of  the  responses  is  shown  in  Figure  9.  This  question  was  set  in  an  inverted  direction: 
more  desirable  responses  are  on  the  left  side  of  the  scale.  Sixteen  students  (53.33%) 
strongly  disagreed  with  the  statements,  i.e.  they  were  not  concerned  about  their  abilities 
to  use  the  web-based  materials  at  all.  However,  five  students  (16.67%)  stated  they  were 
‘somewhat  concerned’,  and  eight  (26.67%)  chose  neutral  response. 


Only  three  correlations  between  this  statement  and  other  variables  were 

significant  (Table  1 1).  I  expected  the  first  two  relations:  students  who  are  more  proficient 

in  using  computers  and  more  skilled  in  searching  on-line  are  less  concerned  about  their 

own  abilities  of  using  web  materials  in  the  course.  The  third  correlation  shows  that  more 

concerned  students  saw  Module  14  material  less  useful  to  them  in  attaining  high 

achievement  in  the  forums  discussions  on  space  control  issues.  The  question  asked 
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students  how  the  material  presented  in  their  module  enabled  them  to  achieve  the 
respective  learning  objectives. 


Figure  9.  Distribution  of  Stated  Concerns  about  Own  Ability  to  Use  Web  Materials 

in  the  Course. 

N  =  30  Mean  =  1 .97  Median  =  1 

Minimum  =  1  Maximum  =  4  Mode  =  1 


Instrument 

Statement 

P 

p  value 

First 

questionnaire 

‘On  a  scale  of  1  to  5,  where  1  =  novice  and  5  = 
expert,  how  would  you  rate  your  current  proficiency 
in  using  computers.’ 

-0.498 

0.005 

First 

questionnaire 

‘On  a  scale  of  1  to  5  where  l=novice  and  5=expert, 
how  would  you  rate  your  on-line  search  resources 
and  techniques?’ 

-0.368 

0.045 

Module  14 

‘Participating  in  forums  (e.g.  discussing,  identifying/ 
consulting  subject  matter  experts),  to  evaluate  space 

-0.364 

0.048 

_ I _ control  issues.’ _ | _ J _ 

Table  1 1 .  Significant  Correlations  and  Levels  of  Significance  between  Concerns  on 

Own  Ability  to  Use  Web  Material  in  the  Course  and  Other  Variables  (Ordered  by  Level 

of  Significance). 
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This  questionnaire  ended  with  an  open-ended  opportunity  for  the  students  to  add 


any  comment:  ‘Any  additional  comments,  concerns,  or  suggestions  are  welcome.’ 


Sixteen  students  (53.33%)  added  comments.  Analyzing  their  responses,  I  grouped 
them  into  four  typical  kinds  of  comments:  Positive  first  impressions,  advice  and 
recommendations,  neutral  comments,  and  concerns'.  Here  are  their  responses: 

Positive  first  impressions 

•  I  like  what  I’ve  seen  so  far! 

•  None  for  now  other  than  site  seems  to  be  very  user  friendly  and  well 
developed. 

•  Looking  forward  to  this  quarter  and  learning  environment. 

•  I’m  looking  forward  to  trying  an  on-line  course! 

•  I  was  a  bit  skeptical,  but  so  far  the  course  seems  to  be  going  well.  The 
thing  I  like  best  is  the  flexibility  - 1  can  read  and  take  the  quizzes  when  it 
is  convenient  for  me.  I  worried  that  I  might  not  keep  up,  but  that  has  not 
been  a  problem. 

•  Great  response  times  to  technical  problems.  The  format  is  very  well 
designed  and  implemented. 

Advice  and  recommendations 

•  Class  sizes  should  be  smaller  or  groups  should  be  assigned  earlier.  The 
general  forum  is  too  large. 

Neutral  comments 

•  None,  at  this  time.  Looking  forward  to  this  first  time  experiment  with  use 
of  online  learning. 

•  None  at  this  time.  Good  Survey! 

•  I  have  no  idea  how  much  time  this  class  will  take. 

•  I  do  prefer  traditional  in  class  teaching.  Online  course  work  requires  more 
effort  because  of  the  all  the  keystroke  required  to  complete  a  course. 
Verbal  communications  allows  the  student  to  quickly  ask  a  question  and 
get  an  answer.  Spell  checking  is  needed  on  these  web  pages  to  help  speed 
up  and  reduce  keystrokes. 
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As  this  is  my  first  web-based  course,  I  am  apprehensive.  I  prefer  a  more 
formal  lecture  style  of  learning.  However,  I  will  give  this  course  my  best 
effort. 


Concerns 

•  I’m  worried  I’ll  spend  more  time  trying  to  figure  out  this  program  then 
learning  about  space. 

•  lam  concerned  about  potential  computer  and  Internet  related  problems. 

•  That  website  access  will  be  available  when  required  and  I  won't  have  to 
continuously  fight  the  system  to  get  access  is  a  concern. 

•  It  seems  that  the  server  is  having  some  capacity  issues  at  peak  times,  i.e. 
during  school  hours.  More  often  than  not  I  am  getting  a  ‘page  not 
available’  image  when  I  click  on  links  or  postings.  Makes  things 
aggravating,  but  not  impossible.  I  can  live  with  it,  but  those  that  chose  #1 
for  questions  #21  and  #22  might  be  hurting.4 


There  were  more  positive  and  neutral  responses  than  negative  comments.  There 
were  also  some  other  ‘signals’  from  the  students  that  deserve  additional  attention: 
comments  that  some  of  them  personally  simply  prefer  classroom  (face-to-face)  learning 
to  on-line  learning;  that  their  concerns  are  about  technology  malfunctions,  not  about  the 
DL  course  itself,  and  that  class  size  and  forums  could  be  too  large. 


First  questionnaire  -  Summary 

This  on-line  questionnaire  was  designed  to  collect  some  general  facts  about  the 
students  and  to  collect  data  the  instructor  and  I  wanted  to  know  from  this  group  of 
students. 

Thirty  students  completed  the  questionnaire.  The  questionnaire  collected  their  e- 
mail  addresses,  data  on  their  status  as  NPS  students,  their  service  (over  43%  were 

4  Answer  #1  to  questions  21  and  22  are  students  who  identified  themselves  as  novices  in  computer 
proficiency  and  in  on-line  search  techniques.  In  this  sample  there  were  no  such  students. 
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Marines,  one  third  were  Navy  officers,  four  were  US  Army,  and  three  others),  and  pay 
grade  (60%  had  0-3,  one  third  0-4,  and  one  0-5).  Pay  grade  had  significant  rank 
correlation  with  previously  experienced  on-line  course  or  courses,  i.e.,  officers  with  a 
higher  pay  grade  (higher  rank)  tended  to  have  more  experience  with  on-line  courses. 
However,  this  was  only  a  mild  correlation. 

The  number  of  enrolled  quarters  varied  from  none  to  six,  with  a  mean  of  3.9.  This 
variable  had  no  significant  correlations  with  any  other  variables,  except  with  three 
student  tracking  variables:  NPS  students  attending  later  quarters  tended  to  visit  the  course 
homepage  fewer  times,  to  send  fewer  posts  to  forums,  and  fewer  follow  up  posts  to  the 
forums. 

Only  two  students  had  some  previous  space-related  duties,  and  only  one  had  taken 
previous  space-related  course.  Three  students  had  experience  with  an  on-line  course, 
including  one  who  attended  an  MBA  program  on-line.  Approximately  half  the  students 
responded  they  would  access  the  course  web  sites  mostly  from  home,  and  another  half 
mostly  from  computer  lab  in  the  campus. 

Four  five-point  scales  about  self-perceived  computer  proficiency,  on-line  search 
skills,  expectations  to  improve  learning,  and  concerns  about  own  abilities  to  use  web- 
based  materials  in  the  course  had  some  interesting  correlations  with  each  other, 
correlations  with  some  data  from  on-line  student  tracking,  and  with  several  items  from 
the  Module  14  questionnaire.  Between  two  and  five  students  thought  they  had  low 
computer  proficiency,  limited  on-line  search  skills,  or  they  had  significant  concerns  about 
their  own  abilities  to  use  web-based  materials.  Responses  to  the  statement  ‘I  expect  this 

online  course  will  be  valuable  and  improve  my  learning’  had  negative  correlation  with 
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some  other  variables:  the  more  a  person  knows  about  computers  and  the  Internet,  the  less 
he/she  expects  to  improve  his/her  learning  in  the  course.  It  is  important  to  note  that 
‘improving  one’s  own  learning’  is  not  same  as  ‘learn  in  a  course’:  the  first  refers  to 
learning  skills  and  habits,  and  the  second  to  knowledge  and  skills  in  a  subject  matter. 

Several  tendencies  can  be  concluded  from  the  student  comments:  their  feedback 
was  more  positive  and  neutral  than  negative;  some  of  them  just  preferred  a  traditional 
classroom  to  an  on-line  course  (without  explicit  explanation).  The  student  concerns  were 
more  about  technical  problems  than  about  the  DL  course  itself.  Some  complained  about 
the  forums  being  too  large. 

2.  Second  Questionnaire  -  ‘A-2’ 

This  questionnaire  (Appendix  B)  was  anonymous.  It  was  administered  after  the 
orientation  part  of  the  course,  i.e.,  during  the  second  and  part  of  the  third  week  of  the 
course.  Thirty-one  students  completed  the  A-2  on-line  questionnaire,  which  means  that 
besides  resident  NPS  students,  one  or  two  of  the  non-resident  students  responded  to  the 
questionnaire.  From  a  total  of  26  questions/statements,  17  were  in  the  form  of  five-point 
scales  (from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’),  and  the  remaining 
questions  were  open-ended. 

The  first  question  was  ‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’  Figure  10  shows  the  distribution  of  the  responses. 

Clearly,  majority  of  the  students  (twenty-seven  out  of  thirty-one  or  87.01%)  did 
not  have  significant  problems  with  setting-up  their  PC.  Only  four  students  said  they  were 
somewhat  uncomfortable. 
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There  were  eight  statistically  significant  rank  correlations  with  other  variables 
(Table  12). 


Figure  10.  Distribution  of  Responses  on  the  Comfort  Level  with  Setting-up  Own 

System. 

N  =  3 1  Mean  =  4.06  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

The  students  who  tended  to  have  a  lower  comfort  level  with  their  PCs  tended  to 
have  a  lower  comfort  level  with  other  technically  demanding  aspects  of  the  course:  using 
forums,  posting  to  forums,  sending  e-mail,  using  word  processing  and  accessing  the 
Internet.  Also,  they  tended  to  see  some  of  the  features  the  instructor  provided  the  students 
to  help  them  with  the  learning  process  as  less  effective  than  the  others:  ‘Rules  of 
engagement’,  course  overview  and  course  syllabus.  ‘Rules  of  engagement’  was  hypertext 
linked  to  the  course  homepage,  with  clearly  stated  expectations  and  standards  the 
instructor  had  about  student  professionalism,  course  participation,  studying,  initiative, 
and  academic  level  of  communication.  The  course  overview,  written  in  the  same  media. 
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briefly  defined  the  objectives  and  organization  of  the  course.  The  content  of  the  course 
syllabus  (also  written  in  hypertext)  was  self-explanatory. 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.692 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.614 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.526 

0.002 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.441 

0.013 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  learn  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.421 

0.018 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.409 

0.022 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.401 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.400 

0.026 

Table  12.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 

Level  in  Setting-up  Own  System  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  second  question  was:  ‘How  would  you  rate  your  comfort  level  at  this  point 
with  accessing  the  Internet?’  The  distribution  of  student  responses  is  in  Figure  11. 
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Comfort  Level  with  Accessing  the  Internet 


Figure  1 1 .  Distribution  of  Responses  on  the  Comfort  Level  with  Accessing  the 

Internet. 

N  =  3 1  Mean  =  4.68  Median  =  5 

Minimum  =  1  Maximum  =  5  Mode  =  5 

Clearly,  the  majority  of  the  students  (25  students  or  80.65%  responded  ‘extremely 
comfortable’)  were  very  comfortable  in  accessing  the  Internet.  Only  one  had  major 
problems,  and  six  were  more  ‘somewhat’  than  ‘extremely’  comfortable.  Obviously,  the 
students  had  relatively  more  problems  in  setting  up  their  systems  and  their  PCs  than  with 
getting  on-line. 

This  variable  showed  statistically  significant  correlations  with  twelve  out  of 
sixteen  other  ordered  categorical  variables  in  the  questionnaire  (Table  13). 

The  highest  correlation  this  variable  had  is,  surprisingly,  with  the  comfort  level  in 
word  processing.  The  students  who  easily  used  word  processing  strongly  tended  to  easily 
access  the  Internet.  The  second  highest  correlation  is  the  association  with  the  comfort 
level  in  using  the  links  feature  in  the  course.  Other  correlations  are  not  so  high,  but  they 
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certainly  indicate  that  the  comfort  level  in  accessing  the  Internet  is  very  indicative  of  the 
overall  comfort  level  a  student  expressed  about  different  aspects  of  the  DL  course. 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.737 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with  the 
links  (i.e.,  additional  web  sites)  feature  being  used  to  teach 
this  course?’ 

0.728 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with  the 
forum  feature  being  used  to  teach  this  course?’ 

0.587 

0.001 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.581 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.571 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.529 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with  the 
email  feature  being  used  to  teach  this  course?’ 

0.477 

0.007 

‘How  effective  were  the  rules  of  engagement?  That  is,  to  what 
extent  did  the  rules  of  engagement  help  you  to  understand 
what  would  be  involved  in  taking  the  Space  Systems 
Technologies  and  Applications  on-line  course?’ 

0.451 

0.011 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.406 

0.023 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.400 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.399 

0.026 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

'  0.398 

0.027 

Table  1 3 .  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  in  Accessing  the  Internet  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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The  word  processing  comfort  level  was  the  next  question:  ‘How  would  you  rate 
your  comfort  level  at  this  point  with  word  processing?’  Figure  12  shows  the  distribution 
of  the  responses. 


Comfort  Level  with  Word  Processing 


Figure  12.  Distribution  of  Responses  on  the  Comfort  Level  with  Word  Processing. 

N  =  3 1  Mean  =  4.65  Median  =  5 

Minimum  =  1  Maximum  =  5  Mode  =  5 

Almost  all  students  were  very  comfortable  with  word  processing  (85.65% 
responded  ‘extremely  comfortable’).  This  is  understandable  because  of  the  common 
usage  of  word  processing  in  modem  office  work.  Only  two  students  had  significant 
problems  with  word  processing. 

The  word  processing  comfort  level  variable  was  significantly  correlated  with  ten 
other  variables  in  the  questionnaire  (Table  14).  Similar  to  accessing  the  Internet  comfort 
level  variable,  this  variable  seems  to  be  very  indicative  of  the  overall  comfort  level  with 
DL  of  a  student  in  this  course.  The  highest  correlations  are  with  the  comfort  level  in 
sending  electronic  messages  and  with  the  comfort  level  in  accessing  the  Internet.  These 
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three  variables  are  probably  key  indicators  of  how  well  a  person  is  able  to  handle  this 
kind  of  course  in  terms  of  the  comfort  level.  Further  analysis  will  show  if  there  are  any 
other  variables  with  similar  significance  in  this  instrument. 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?  • 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.737 

<  0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.541 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.560 

0.004 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.450 

0.011 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course?’ 

0.425 

0.017 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  learn  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.425 

0.017 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.409 

0.022 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.388 

0.031 

Table  14.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  Word  Processing  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  fourth  question  addressed  the  forums:  ‘How  would  you  rate  your  comfort 
level  at  this  point  with  posting  to  file  forum?’  Figure  13  shows  the  distribution  of  the 
responses. 
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Comfort  Level  with  Posting  to  Forums 
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Figure  13.  Distribution  of  Responses  on  the  Comfort  Level  with  Posting  to  the 

Forums. 

N  =  3 1  Mean  =  4.00  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

A  large  majority  (26  students  or  83.87%)  of  students  felt  comfortable  with  posting 
to  the  forums,  but  more  of  them  were  ‘somewhat  comfortable’  (16  students  or  51.61%) 
than  ‘extremely  comfortable’  (10  students  or  32.26%).  Four  students  (12.90%)  felt 
uncomfortable,  and  one  was  neutral  about  this  issue. 

Table  15  shows  significant  correlations.  Two  variables  about  the  comfort  level 
with  two  aspects  of  the  forum  used  in  the  course  are  very  highly  correlated.  Other 
correlations  confirm  that  this  variable  is  one  of  those  central  comfort  level  variables 
within  a  DL  environment. 

The  fifth  question  was:  ‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’  Figure  14  shows  the  distribution. 
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Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.800 

<  0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.692 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.571 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.537 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.506 

0.004 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail? 

0.491 

0.005 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  through  your 
Internet  service  provider  (ISP)?’ 

0.489 

0.005 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.464 

0.009 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.456 

0.010 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  learn  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.445 

0.012 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.422 

0.018 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  on  CD-ROM?’ 

0.410 

0.022 

Table  15.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  Posting  to  the  Forums  and  Other  Variables  (Ordered  by  Level  of 

Significance). 


Twenty-five  students  (80.65%)  felt  extremely  comfortable  in  sending  e-mail, 
which  is  to  be  expected  in  the  NPS  environment  since  every  student  is  expected  to  check 
e-mail  frequently.  Within  the  course  context,  e-mail  was  functioning  well  as  one  of  the 
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key  ways  of  communicating.  However,  few  students  had  problems  with  sending  e-mail: 
two  (6.45%)  felt  ‘extremely  uncomfortable’  and  one  (3.23%)  ‘somewhat  uncomfortable’. 


Comfort  Level  with  Sending  E-mail 


Figure  14.  Distribution  of  Responses  on  the  Comfort  Level  with  Sending  E-mail. 

N  =  3 1  Mean  =  4.55  Median  =  5 

Minimum  =  1  Maximum  =  5  Mode  =  5 

This  variable  had  ten  significant  correlations  (Table  16). 

The  most  significant  is  the  correlation  between  the  comfort  level  with  sending  e- 
mail  and  with  word-processing.  Other  correlations,  not  as  high  as  the  first  one,  are  with 
links  usage,  accessing  the  Internet,  setting  up  the  system,  etc.  This  is  the  fifth  of  those 
‘core’  comfort  level  variables. 
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Question 

. .  P . 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.780 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.540 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.529 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.526 

0.002 

How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum? 

0.491 

0.005 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.439 

0.013 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  leam  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.434 

0.015 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  leam  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.415 

0.020 

How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course? 

0.410 

0.022 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.392 

0.029 

Table  16.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  Sending  E-mail  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  next  question  was  open-ended.  It  asked:  ‘In  what  areas  of  the  on-line 
technologies  (system  set-up,  accessing  the  internet,  word-processing,  posting  to  forums) 
do  you  still  need  help?’  Twenty  students  answered  ‘None’  and  one  who  answered  ‘None, 
just  practice.’  I  divided  the  other  responses  into  two  groups:  ‘Forum  postings’  and 
‘Other’: 
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Forum  Postings 

•  Post  or  replying  to  a  comment  in  the  Forum  is  still  less  than  user  friendly 

•  Posting  to  Forums 

•  The  postings  are  still  rough  as  well  as  the  quizzes 

•  Posting  to  the  forum  is  still  hit  or  miss.  In  other  words,  I'm  not  confident 
that  what  I  submit  is  going  where  I  think  I  am  sending  it. 

•  Posting  without  loosing  work 

•  lam  still  getting  run-time  script  errors  when  accessing  postings 

Other 

•  System  set-up:  advanced  issues,  troubleshooting 

•  Very  confusing  because  of  all  bugs  in  this  system.  Thanks  for  the 
instructor’s  summary  via  email  -  this  helped  me  stay  on  track. 

•  At  this  time  was  changing  the  subject  type,  but  that  has  all  been  resolved.5 

To  conclude,  the  students’  answers  to  this  question  indicated  that  problems  with 
forum  postings  were  most  common,  but  the  majority  of  them  did  not  have  significant 
problems  with  on-line  technology  that  had  been  not  solved  when  they  answered  the 
questionnaire. 

The  sixth  comfort  level  scale  was  about  software  technical  support:  ‘How  would 
you  rate  your  comfort  level  at  this  point  with  the  technical  support  provided  for  using  the 
WebCT  software?’  The  distribution  of  the  student  responses  can  be  seen  in  Figure  15. 


5  This  response  is  probably  also  related  to  forum  posting  issues. 
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Figure  1 5 .  Distribution  of  Responses  on  Comfort  Level  with  the  Technical  Support 

Provided  for  the  WebCT  Software. 

N  =  31  Mean  =  3.61  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

A  relatively  large  number  of  neutral  answers  (twelve  or  38.71%)  indicates  many 
students  did  not  need  to  use  the  technical  support  for  the  WebCT  software.  Those  who 
needed  that  support  were  mostly  ‘somewhat’  or  ‘extremely’  comfortable,  i.e.,  satisfied 
with  the  support  provided.  Only  three  students  were  ‘somewhat  uncomfortable’,  i.e.,  not 
satisfied  with  the  technical  support. 

This  variable  is  significantly  correlated  only  with  three  other  variables  in  the 
instrument  (Table  17). 
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Question 

P 

p  value 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  to  understand 
what  you  would  leam  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.462 

0.009 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.408 

0.023 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course?’ 

0.406 

0.023 

Table  1 7.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 

Level  with  the  Technical  Support  Provided  for  the  WebCT  Software  and  Other  Variables 

(Ordered  by  Level  of  Significance). 

The  students  more  satisfied  with  the  technical  software  support  tended  to  estimate 
some  course  features  set  by  the  instructor  (module  objectives  and  rules  of  engagement)  as 
more  effective  (i.e.,  they  had  more  utility  and  fewer  difficulties  with  these  features).  The 
third  relation  shows  that  lower  comfort  level  with  technical  software  support  was 
associated  with  lower  comfort  level  with  e-mail  features  in  the  course. 

The  tenth  variable  in  the  instrument  asked  the  students  the  following:  ‘How  would 
you  rate  your  comfort  level  at  this  point  with  the  technical  support  available  to  you 
through  your  Internet  service  provider  (ISP)?’  Figure  16  below  shows  their  responses. 
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Figure  16.  Distribution  of  Responses  on  Comfort  Level  with  the  Technical  Support 

Provided  by  the  ISP. 

N  =  3 1  Mean  =  3.74  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  3 

This  distribution  was  very  similar  to  the  responses  to  the  WebCT  technical 

support  question,  with  a  slight  difference.  There  were  more  neutral  answers  (fourteen  or 

45.16%  did  not  need  any  help  from  their  ISP)  and  only  one  uncomfortable  student.  This 

variable  had  significant  correlation  with  three  other  variables  (Table  18). 
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Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.489 

0.005 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.399 

0.026 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  learn  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.395 

0.028 

Table  1 8.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Technical  Support  Provided  by  the  ISP  and  Other  Variables  (Ordered  by 

Level  of  Significance). 

Found  correlations  are  statistically  significant,  but  mild.  Students  comfortable 
with  ISP  support  tended  to  feel  more  comfortable  with  posting  e-mail  and  with  the  course 
features  of  the  syllabus  and  course  overview. 


The  following  question  was  about  student  comfort  level  with  automated  data 
processing  (ADP)  office  technical  support.  The  responses  were  distributed  as  follows  in 
Figure  17.  Eighteen  responses  (58.06%)  were  neutral.  It  seems  students  did  not  have  to 
rely  very  much  on  these  support  tools.  Only  two  students  (6.45%)  students  said  they  were 
‘somewhat  uncomfortable’. 


Only  one  variable  was  statistically  significantly  correlated  with  this  variable.  It 
was  the  comfort  level  with  the  course  overview  feature  (‘How  effective  was  the  course 
overview?  That  is,  to  what  extent  did  the  course  overview  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and  Applications  on-line  course?’). 
Spearman’s  coefficient  of  rank  correlation  was  p  =  0.405,  and  the  significance  level  p  = 
0.024. 
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Figure  17.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Technical 

Support  Provided  by  the  ADP  Office, 

N  =  31  Mean  =  3.35  Median  =  3 

Minimum  =  2  Maximum  =  5  Mode  =  3 

The  instructor  was  also  interested  in  the  student  comfort  level  with  the  ‘Pegasus’ 

CD-ROM  provided  by  CFU.  The  question  was:  ‘How  would  you  rate  your  comfort  level 

at  this  point  with  the  technical  support  available  to  you  on  CD-ROM?’  Figure  18  shows 

how  the  student  responses  were  distributed. 


Comfort  Level  with  'Pegasus'  CD-ROM 
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Figure  1 8.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Technical 

Support  Provided  with  the  CD-ROM. 
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N  =  31  Mean  =  3.19  Median  =  3 

Minimum  =  2  Maximum  =  5  Mode  =  3 

Most  students  (25  students  or  80.65%)  responded  neutrally  to  this  question.  Only 
six  responses  were  non-neutral,  and  one  of  these  six  was  negative  (‘somewhat 
uncomfortable’).  However,  four  correlations  between  this  and  other  variables  were 
statistically  significant  (Table  19). 


Question 

P 

m 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.472 

0.007 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.410 

0.022 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.392 

0.029 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.362 

0.045 

Table  19.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  CD-ROM  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  first  correlation  tells  us  that  same  students  who  had  a  lower  comfort  level 
with  the  CD-ROM  tended  to  have  a  lower  comfort  level  with  forums  in  general;  with 
posting  to  forums,  and  with  the  links  feature.  Also,  they  tended  to  see  rules  of 
engagement  less  effectively  than  other  students. 

Next  was  the  open-ended  question  about  other  technical  problems.  Its  text  was: 
‘What  technical  issues  (e.g.,  those  related  to  WebCT,  Internet  Services,  Local  ADP,  CD- 
ROM,  etc.)  do  you  still  need  to  resolve?’  Nineteen  students  responded  ‘None’.  I  grouped 
the  rest  of  responses  into  three  main  groups: 
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Quiz 


•  The  grading  problem  with  the  quizzes 

•  The  only  issue  is  problem  with  the  quizzes.  However,  I  know  that's  being 
worked 

•  It's  frustrating  that  the  quiz  portion  is  still  not  functioning  properly  -  I  do 
not  want  to  get  too  far  ahead  into  the  course  and  then  have  to  back  up  and 
remediate  several  quizzes  at  one  time  once  it  get  fixed 

I  Haven’t  Started  Yet6 

•  Don’t  know  yet! 

•  Haven't  had  to  use  any 

•  All  answers  should  be  'n/a'  because  I  have  yet  to  use  them 

Miscellaneous 

•  The  inability  to  use  certain  links  from  the  course  schedule,  and  the 
problems  with  scoring  quizzes 

•  Some  of  the  questions  do  not  pertain  in  Qll.  ‘N/a’  should  be  given  as  a 
response  option7 

•  Just  glitches  everyone  else  is  having  (with  WebCT) 

•  None  other  links  than  those  that  are  group  problems.  (Not  being  able  to 
take  the  quizzes  and  the  schedule  links.) 

•  Problems  logging  on  from  AOL 

•  What  CD?8 

After  this,  there  was  another  open-ended  question  about  any  other  possible 
thoughts  on  this  aspect  of  the  course  (‘Please  provide  any  additional  thoughts.’).  Twenty- 
seven  students  did  not  have  any  comments  and  four  students  did.  Here  are  the  comments: 


6  This  means  that  these  students  didn’t  have  opportunity  to  face  any  technical  problems  yet. 

7  Comment  about  this  questionnaire. 

8  This  student  didn’t  have  any  information  about  ‘Pegasus’  CD-ROM. 
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Negative  Comment 


•  In  general,  tech  support  sucks.  Most  of  these  people  can't  think  beyond  the 
canned  answers  for  them  within  a  computer  thus  making  the  support  they 
can  provide  you  minimal  and,  ultimately,  useless.  Which  is  not  to  say  that 
good  tech  support  is  not  available.  Just  that  it's  not  available  for  the 
categories  you  listed  and  the  companies  that  provide  them  (Pacific  Bell, 
NPS). 

Neutral  Comments 

•  The  WebCT  interface  still  needs  some  work 

•  The  problems  that  I  usually  encounter  are  the  same  that  other  are 
encountering 

Positive  Comment 

•  Help  has  been  great! 


The  next  eight  five-point  scale  questions  asked  the  students  how  comfortable  they 
were  with  different  features  used  in  the  course:  forums,  e-mail,  rules  of  engagement,  etc. 

The  first  question  was  about  the  forum  feature:  ‘How  would  you  rate  your 
comfort  level  at  this  point  with  the  forum  feature  being  used  to  teach  this  course?’  Figure 
19  shows  the  distribution. 

An  overwhelming  majority  of  the  students  was  feeling  ‘somewhat  comfortable’ 

(17  students  or  54.84%)  or  ‘extremely  comfortable’  (seven  students  or  22.58%)  with  the 

$ 

forums.  Only  four  (12.90%)  felt  uncomfortable  (one  of  them  ‘extremely  uncomfortable’), 
and  three  other  students  were  neutral.  Interestingly,  their  responses  to  this  question  are 
similar  but  not  identical  to  the  question  about  the  comfort  level  with  posting  to  forums 
(Figure  13).  Briefly,  more  of  them  were  ‘extremely  comfortable’  with  posting  to  the 
forums  than  with  the  use  of  the  forums  in  the  course. 
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Comfort  Level  with  Forums  Use  in  the  Course 


Figure  19.  Distribution  of  Responses  on  the  Forums  Used  in  the  Course. 


N  =  31  Mean  =  3.84  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

The  calculation  showed  that  ten  rank  correlations  between  this  and  other  variables 

were  statistically  significant  (Table  20). 


The  highest  correlation  is  between  two  variables  asking  about  the  comfort  level 
with  posting  to  the  forums  and  about  the  comfort  level  with  the  use  of  forums  in  the 
course.  Setting-up  the  systems,  accessing  the  Internet,  using  word-processing,  and 
comfort  with  sending  e-mail  appeared  to  be  important  issues  correlated  with  the  comfort 
of  use  of  the  forums  and  with  a  few  other  variables  in  this  set. 
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Question 

P 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.800 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.614 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.587 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.503 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course?’ 

0.491 

0.005 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  on  CD-ROM?’ 

0.472 

0.007 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.406 

0.023 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.392 

0.029 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.360 

0.047 

Table  20.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  Use  of  the  Forums  in  the  Course  and  Other  Variables  (Ordered  by  Level  of 

Significance). 

Next  was  the  question  about  the  comfort  level  with  the  use  of  e-mail  features  in 
the  course  (Figure  20). 
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Figure  20.  Distribution  of  Responses  on  the  Comfort  Level  with  the  E-mail  Feature 

in  the  Course. 
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N  =  31  Mean  =  3.98  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  5 

There  is  a  very  obvious  difference  between  this  distribution  of  responses  and  the 
responses  to  the  question  about  the  comfort  level  with  sending  e-mail  (Figure  14),  where 
over  26  students  (83.87%)  responded  to  being  ‘extremely  comfortable’.  Here  only 
thirteen  students  (41.94%)  chose  the  same  response.  It  is  not  clear  why  so  many  students 
(nine  or  29.03%)  chose  a  neutral  answer.  It  could  be  because  of  the  frequency,  content,  or 
other  characteristics  of  e-mail  messages  they  were  receiving  from  the  instructor,  or  they 
did  not  quite  understand  the  question.  Table  21  shows  the  correlations  between  this  and 
other  variables.  Correlations  are  moderate. 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.491 

0.005 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  to  understand 
what  you  would  learn  in  die  Space  Systems  Technologies  • 
and  Applications  on-line  course?’ 

0.491 

0.005 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.477 

0.007 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  to  understand 
what  you  would  learn  in  the  Space  Systems  Technologies 
and  Applications  on-line  course?’ 

0.436 

0.014 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.425 

0.017 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.410 

0.022 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  provided  for  using  the  WebCT 
software?’ 

0.406 

Table  21 .  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 

Level  with  the  E-mail  Feature  in  the  Course  and  Other  Variables  (Ordered  by  Level  of 

Significance). 
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As  in  many  other  similar  cases,  several  different  variables  with  the  comfort  level 
answers  about  various  aspects  of  the  course  (forums,  accessing  the  Internet,  word 
processing)  and  course  features  (course  overview,  module  objectives)  are  correlated  with 
medium  to  lower-medium  coefficients9. 

The  following  question  asked  students  about  the  quiz  feature:  ‘How  would  you 
rate  your  comfort  level  at  this  point  with  the  quiz  feature  being  used  to  teach  this  course?’ 
Figure  21  shows  the  distribution  of  the  responses. 

Interestingly,  in  the  distribution  of  the  student  responses  students  split  into  two 
different  groups:  approximately  one  third  of  the  students  (ten  students  or  32.26%)  said 
they  felt  uncomfortable  with  the  quiz  feature  (half  of  them  ‘somewhat’,  another  half 
‘extremely  uncomfortable’),  and  two  thirds  (nineteen  students  or  61.29%)  comfortable. 
Only  two  students  chose  a  neutral  response.  The  fact  that  during  the  course  there  were 
some  technical  problems  with  the  quizzes  only  partially  explains  such  a  distribution. 
Also,  the  instructor’s  notes  suggest  that  the  students  had  different  interpretations  of  the 
importance  of  the  quizzes  in  determining  their  final  grade.  This  phenomenon  will  be 
examined  in  the  section  on  interviews  with  students. 

This  variable  has  statistically  significant  rank  correlations  with  six  other  variables 
in  the  instrument  (Table  22). 


9  Statisticians  usually  consider  correlations  over  0.75  or  0.80  as  ‘high’  and  those  below  0.50  or  0.45  as 
‘low’.  They  do  not  take  into  consideration  at  all  any  correlation  if  it  is  not  statistically  significant  with  at 
least  a  previously  determined  level  of  significance. 
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Figure  21.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Quiz  Feature  in 

the  Course. 

N  =  31  Mean  =  3.45  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  5 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.537 

0.002 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.531 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.401 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.399 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.388 

0.031 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.360 

0.047 

Table  22.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Quiz  Feature  in  the  Course  and  Other  Variables  (Ordered  by  Level  of 

Significance). 
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The  next  question  was:  ‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to  teach  this  course?’  The 
distribution  of  the  responses  is  shown  in  Figure  22. 
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Figure  22.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Links  Feature  in 

the  Course. 

N  =  31  Mean  =  4.45  '  Median  =  5 

Minimum  =  1  Maximum  =  5  Mode  =  5 

The  vast  majority  of  the  students  (twenty-one  students  or  67.74%)  were 
‘extremely  comfortable’  with  links  as  an  important  component  of  the  on-line  course 
material.  Less  (seven  or  22.58%)  were  ‘somewhat  comfortable’,  and  only  two  of  them 
indicated  they  felt  uncomfortable.  Of  all  the  course  features  we  analyzed  so  far  in  this 
chapter,  links  clearly  had  the  highest  comfort  level  among  the  students. 

Table  23  lists  eleven  variables  that  have  statistically  significant  rank  correlations 
with  the  links-related  comfort  level.  The  comfort  level  with  the  links  feature  highly 
correlates  with  the  comfort  level  in  accessing  the  Internet,  and  the  correlations  are 
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somewhat  lower  with  the  comfort  level  with  word  processing,  sending  e-mail,  and  other 
on-line  course  features  like  rules  of  engagement  and  module  objectives. 

Another  element  of  on-line  course  materials  was  the  course  overview.  The 
question  was:  ‘How  effective  was  the  course  overview?  That  is,  to  what  extent  did  the 
course  overview  help  you  to  understand  what  you  would  learn  in  the  Space  Systems 
Technologies  and  Applications  on-line  course?’  Figure  23  shows  the  distribution  of  the 
responses. 


Question 

P 

p  value 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.728 

<  0.001 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to 
understand  what  would  be  involved  in  taking  the  Space 
Systems  Technologies  and  Applications  on-line  course?’ 

0.629 

<  0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.541 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.540 

0.002 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  to  understand  what 
you  would  learn  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.524 

0.002 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  to  understand  what 
you  would  leam  in  the  Space  Systems  Technologies  and 
Applications  on-line  course?’ 

0.529 

0.003 

i  raavis.K: *  *  i  rax* 

0.499 

0.004 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.464 

0.009 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.406 

0.023 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.401 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  on  CD-ROM?’ 

0.362 

0.045 

Table  23.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Link  Feature  in  the  Course  and  Other  Variables  (Ordered  by  Level  of 

Significance). 
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Figure  23.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Course 

Overview. 


N  =  31  Mean  =  4.16  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

Only  three  students  (9.67%)  felt  ‘somewhat  uncomfortable’  with  the  course 
overview,  and  three  students  (9.67%)  chose  neutral  statements.  Most  of  the  students  (14 
or  45.16%)  felt  ‘somewhat  comfortable’,  and  the  remaining  students  (12  or  38.71%)  even 
‘extremely  comfortable’,  which  gives  very  encouraging  feedback  to  the  course  instructor. 
The  correlations  between  this  and  other  variables  are  shown  in  Table  24. 

These  correlations  clearly  show  a  significant  level  of  association  between  all 
additional  on-line  course  content  created  by  the  instructor  to  facilitate  student  learning: 
course  overview,  rules  of  engagement,  module  objectives,  and  course  syllabus.  In  other 
words,  the  same  students  who  felt  uncomfortable  with  any  of  these  textual  features  used 
in  the  course,  tended  to  have  a  similar  impression  about  some  of  the  other  content  areas 
in  the  course.  Other  variables,  those  reflecting  ‘core  on-line  issues’,  such  as  the  comfort 
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level  with  accessing  the  Internet  or  with  word  processing,  and  some  others  such  as  the 
comfort  level  with  ADP  office  support,  were  significant. 


Question 

P 

p  value 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  (. . .)?’ 

0.621 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.581 

0.001 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  (...)?’ 

0.569 

0.001 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  (...)?’ 

0.515 

0.003 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.499 

0.004 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.445 

0.012 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course?’ 

0.436 

0.014 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.424 

0.017 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.421 

0.018 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.415 

0.020 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  through  the  school 
ADP  office?’ 

0.405 

0.024 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  through  your 
Internet  service  provider  (ISP)?’ 

0.395 

0.028 

Table  24.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Course  Overview  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  next  item  was:  ‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  to  understand  what  you  would  learn  in  the 
Space  Systems  Technologies  and  Applications  on-line  course?’  The  distribution  of  the 
student  responses  is  shown  in  Figure  24. 


77 


Comfort  Level  with  Module  Objectives 


N  =  3 1  Mean  =  4.06  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

This  distribution  is  very  similar  to  the  distribution  of  the  responses  on  the  comfort 
level  with  the  course  overview  (Figure  23),  except  there  were  somewhat  more  neutral 
responses.  Only  one  student  felt  somewhat  uncomfortable  with  module  objectives. 

Eight  correlations  were  significant.  Similar  comments  apply  to  this  question  about 
comfort  level  with  module  objectives  as  with  the  previous  one:  correlations  between 
additional  on-line  course  contents  (‘course  overview’  and  ‘course  syllabus)  and  ‘core  on¬ 
line  issues’  dominate  (Table  25).  A  similar  tendency  can  be  expected  with  responses  to 
the  next  two  questions  also. 
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Question 

P 

P  value 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  (. . .)?’ 

0.565 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.524 

0.002 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  (...)?’ 

0.523 

0.003 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  (. . .)?’ 

0.515 

0.003 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  email  feature  being  used  to  teach  this  course?’ 

0.491 

0.005 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  forum  feature  being  used  to  teach  this  course?’ 

0.473 

0.007 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  provided  for  using  the  WebCT 
software?’ 

0.462 

0.009 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.406 

0.023 

Table  25.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Module  Objectives  and  Other  Variables  (Ordered  by  Level  of 

Significance). 

The  next  question  was:  “How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  to  understand  what  would  be  involved 
in  taking  the  ‘Space  Systems  -  Technologies  and  Applications’  on-line  course?”  The 
distribution  of  the  student  responses  is  in  Figure  25. 
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Figure  25.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Rules  of 

Engagement. 


N  —  3 1  Mean  =  4.19  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 

This  distribution  follows  a  similar  pattern  as  the  previous  two.  There  were  more 
‘somewhat  comfortable’  responses:  51.61%  to  this  question  compared  to  the  previous 
two  questions  (45.16%).  Only  one  student  expressed  feeling  uncomfortable  with  the  rules 
of  engagement  in  the  course. 

Table  26  shows  all  significant  correlations  between  this  and  the  other  variables. 
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Question 

P 

p  value 

‘How  effective  was  the  course  syllabus?  That  is,  to  what 
extent  did  the  course  syllabus  help  you  (. . .)?’ 

0.718 

<0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.629 

<  0.001 

0.621 

<0.001 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  (. . .)?’ 

0.565 

0.001 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
quiz  feature  being  used  to  teach  this  course?’ 

0.531 

0.002 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.456 

0.010 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.451 

0.011 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
word  processing?’ 

0.450 

0.011 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.441 

0.013 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.439 

0.013 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  provided  for  using  the  WebCT 
software?’ 

0.408 

0.023 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  on  CD-ROM?’ 

.  0.392 

0.029 

Table  26.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Rules  of  Engagement  and  Other  Variables  (Ordered  by  Level  of 

Significance). 


Correlations  followed  a  very  similar  pattern  as  discussed  above. 

The  last  five-point  scale  question  in  this  instrument  was:  “How  effective  was  the 
course  syllabus?  That  is,  to  what  extent  did  the  course  syllabus  help  yOu  to  understand 
what  you  would  learn  in  the  ‘Space  Systems  Technologies  and  Applications’  on-line 
course?”  Figure  26  shows  the  distribution  of  the  responses. 
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Comfort  Level  with  Course  Syllabus 


Figure  26.  Distribution  of  Responses  on  the  Comfort  Level  with  the  Course  Syllabus. 

N  =  31  Mean  =  3.84  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

Three  students  (9.68%)  felt  ‘uncomfortable’  about  the  course  syllabus,  and  seven 
(22.58%)  chose  a  neutral  response.  The  latter  students  probably  did  not  carefully  examine 
the  syllabus.  Twenty-two  (70.97%)  felt  ‘comfortable’  (almost  equally  divided  between 
‘somewhat’  and  ‘extremely  comfortable’.  Correlations  formed  a  similar  pattern  for  the 
comfort  level  with  the  course  overview,  module  objectives,  and  rules  of  engagement. 
Table  27  shows  coefficients  and  significance  levels. 

These  correlations  also  followed  the  expected  pattern. 

At  the  end  of  the  questionnaire,  there  were  five  open-ended  questions  asking  the 
students  about  needed  help  with  the  web-based  features,  general  liking  of  the  course, 
concerns,  suggestions,  and  question  on  how  they  would  create  their  own  on-line  course  if 
they  were  instructors. 
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Question 

P 

p  value 

‘How  effective  were  the  rules  of  engagement?  That  is,  to 
what  extent  did  the  rules  of  engagement  help  you  (. . .)?’ 

0.718 

<0.001 

‘How  effective  was  the  course  overview?  That  is,  to  what 
extent  did  the  course  overview  help  you  (...)?’ 

0.569 

0.001 

‘How  effective  were  module  objectives?  That  is,  to  what 
extent  did  the  module  objectives  help  you  (. . .)?’ 

0.523 

0.003 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  links  (i.e.,  additional  web  sites)  feature  being  used  to 
teach  this  course?’ 

0.509 

0.003 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
sending  electronic  mail?’ 

0.434 

0.015 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  to  the  forum?’ 

0.422 

0.018 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
setting-up  your  system?’ 

0.401 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
the  technical  support  available  to  you  through  your 
Internet  service  provider  (ISP)?’ 

0.399 

0.026 

‘How  would  you  rate  your  comfort  level  at  this  point  with 
accessing  the  Internet?’ 

0.398 

0.027 

Table  27.  Significant  Correlations  and  Levels  of  Significance  between  the  Comfort 
Level  with  the  Course  Syllabus  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  first  of  these  open-ended  questions  was  the  question  about  help  the  students 
might  need:  ‘In  what  areas  do  you  still  need  help  with  the  web-based  technological 
features  (forum,  email,  quizzes,  links,  etc)  used  to  teach  this  on-line  course?’  Twenty 
students  answered  they  needed  no  such  help.  Here  are  the  other  answers,  systematized: 

Quiz 

•  None  -  just  fix  the  quiz  section 

•  Fix  quiz  2,  get  quiz  3  up  and  running 

•  I  await  resolution  of  the  quiz  technical  difficulties 

•  The  quizzes  obviously  need  to  be  repaired  technically 

•  Quizzes  -  we  just  want  die  site  fixed 

•  Quizzes  need  a  summary  which  questions  are  wrong  and  unanswered 
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Other  and  unclear 

•  Forum  posting 

•  There  are  still  some  minor  bugs 

Neutral 

•  I  haven’t  used  e-mail.  Quiz  grading  is  much  better 

The  responses  clearly  suggest  that  at  the  moment  of  responding  to  this  on-line 
questionnaire,  the  only  significant  technical  problem  that  the  students  needed  help  with 
was  the  ‘bug’  in  quiz  scoring  (six  of  the  ten  responses  addressed  this  issue). 

The  second  question  asked  students  if  they  liked  the  course:  ‘So  far,  what  do  you 
like  about  this  on-line  course?’  Here  is  how  the  students  responded: 

Time  management  flexibility  and  accessibility10 

•  Accessibility 

•  Can  work  on  it  at  my  leisure 

•  Convenience  to  do  it  on  my  schedule 

•  Ease  of  access. . .  Able  to  access  around  my  schedule 

•  Flexibility  of  my  schedule  -  ability  to  review  -  additional  web  links  to 
areas  of  interest 

•  Flexibility!  [Three  answers] 

•  Freedom  to  work  at  own  pace 

•  I  can  do  it  anytime 

•  I  like  the  pace.  However,  I'm  not  sure  if  I'm  learning  more  or  less  than 
what  is  expected. 

•  Self  paced  with  visible  objectives  and  requirements 

•  Self-paced.  You  can  work  it  into  your  schedule  easier 


10  These  two  types  of  answers  were  combined  together  because,  according  to  student  responses,  they 
seem  to  be  very  close  to  each  other  within  the  context  of  die  on-line  course. 
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•  The  ability  to  manage  your  own  time 

•  The  convenience  of  managing  my  time  as  it  relates  to  the  requirements  of 
the  class.  I  can  spend  extra  time  or  less  time  is  required. 

•  The  convenience  of  the  course,  i.e.  being  able  to  do  die  course  whenever  I 
choose 

•  The  flexibility  to  learn  when  I  am  ready  to 

•  The  freedom  of  time... 

•  The  autonomy.  I  like  doing  the  work  on  my  own  so  I  can  skim  what  I 
already  know  and  go  deeper  on  things  that  interest  me. 

Forums 

•  The  forum  interaction  is  great,  but  I  would  agree  with  the  posting  that  we 
should  meet  once  a  week  or  so  at  the  later  stages  of  the  course  to  have 
question/answer  periods  with  the  instructor 


Multiple  answers  and  miscellaneous 

•  I  can  work  on  it  all  hours  and  at  my  home  PC.  Also  the  continuous 
postings  to  the  forums  are  beneficial. 

•  I  like  the  forums  format.  I  also  like  the  links  in  the  modules 

•  I  like  the  layout  and  interface  real  well.  Although  the  text  is  extremely 
extensive,  it  has  been  informative  and  enjoyable  to  read  so  far. 

•  I  like  the  links  and  forum  feature 

•  I  like  the  links  to  sites  that  I  may  not  have  found  otherwise.  The  book 
seemed  a  tad  bit  too  simple  at  first,  but  as  I  got  to  the  orbit  section,  there  is 
some  in-depth  material. 

•  Multiple-choice  open  book  repeatable  quizzes 

•  Still  working 

•  The  ability  to  crawl  out  of  the  rack  and  check  the  forums,  and  make 

comments.  The  ability  to  do  the  work  to  better  fit  my  schedule. 

•  The  text  and  the  links 

•  Very  insightful  discussions  in  the  forum;  excellent  support  for  technical 
issues;  a  very  responsive  instructor;  and  links  to  very  interesting  web  sites 


Clearly,  the  students  outlined  time  flexibility  and  accessibility  of  the  course 

material  whenever  it  suits  them  as  main  reason  for  liking  the  course  (19  out  of  30 
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responses  or  63.33%  of  the  responses)  The  same  issues  are  also  mentioned  in  three  of  the 
answers  in  the  third  group  (Multiple  Answers  and  Miscellaneous). 


The  next  question  asked  students  about  concerns  they  had  about  the  course:  ‘So 
far,  what  are  your  concerns  about  this  on-line  course?’  Eleven  students  responded  ‘None’ 
or  did  not  respond  at  all  which  implies  they  had  no  significant  concerns.  Other  responses 
are  systematized  as  follows: 


Forum 

•  Too  many  forum  postings!  It’s  just  too  difficult  and  time  consuming  to 
realistically  read  everything  that  everyone  posts.  Forum  groups  need  to  be 
broken  down  (say,  5-7  people)  so  that  we  can  share  ideas  in  a  smaller 
group.  Also,  the  fact  that  we  are  getting  behind  with  Modules  that  don't 
work  is  frustrating.  It’s  as  if  we’re  at  a  standstill  and  will  need  to  catch  up 
later  on. 

Comparison  with  face-to-face  class 

•  Depth  of  knowledge  vs.  in-class  instruction 

•  I  am  concerned  that  I  may  not  be  getting  the  same  information  as  my 
counterparts  in  the  in-room  class 

•  See  above  #25.  I’ve  learned  some  things...  Not  sure  if  I’m  getting  the 
same  value  as  students  who  are  actually  sitting  in  a  classroom  with  a 
professor. 

Learning  pace  and  organization 

•  Getting  too  far  ahead  in  the  assignments  without  being  able  to  keep  up 
with  the  quizzes  because  that  section  is  not  working  properly 

•  Keeping  up  on  the  readings  and  not  falling  behind 

•  My  personal  lack  of  urgency  for  the  first  couple  of  weeks.  And  my 
unwillingness  to  read  something  posted  over  a  paragraph  or  two  in  length. 

Technical  problems 

•  Run-time  errors...  They  are  not  due  to  my  memory  allocation  because  I'm 
getting  them  no  matter  what  computer  I’m  accessing  the  web  site  from... 
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•  The  apparent  slowness  with  which  the  WebCT  has  been  able  to  resolve  the 
technical  problems  associated  with  the  quizzes 

•  The  password  and  user  id  issue,  but  that  has  been  addressed 

•  Too  many  problems  up  front  that  should  have  been  worked  out  ahead  of 
time 

Grading 

•  How  will  we  be  graded? 

•  I'm  concerned  with  how  we  will  be  evaluated  given  that  we're  having 
problems  with  scoring  the  quizzes. 

•  That  some  wordy  students  will  get  better  grades,  even  when  the  context  is 
not  worthy.  I  hope  that  the  grade  is  based  on  the  quality  of  the  content,  not 
the  quantity.  A  lot  of  the  comments  are  verbose  and  boring. 

Multiple  answers  and  miscellaneous 

•  My  only  concern  so  far  is  about  missing  an  event.  Since  some  of  the  items 
are  not  working,  I  may  overlook  a  part  this  is  working. 

•  The  lack  of  interaction  with  the  professor 

•  Too  cumbersome  to  always  check  for  replies...  Also  sometimes  ready  to 
take  quiz  or  evaluation  and  it  has  not  been  posted  yet. 

Positive  opinions 

•  No  negative  concerns,  I  am  very  positive  and  motivated  to  continue  into 
deep  space! 

•  That  there  will  be  no  more  after  this.  It  is  a  great  medium  for  learning 

There  was  no  one  predominant  issue  students  addressed,  but  students  expressed 
concerns  about  different  themes.  Some  responses  seem  to  be  unrelated  to  the  specific  DL 
nature  of  the  course,  but  were  more  general  student  concerns  (learning  pace  and 
organization)  that  appear  in  any  type  of  class.  Student  concerns  are  grouped  around  the 
following  issues:  technical  problems  (four  comments);  uncertainty  whether  the 
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possibilities  of  DL  are  equal  to  those  is  face-to-face  class  (three);  grading  (three);  forums 
(one)  and  mixed/miscellaneous  problems  (three  comments). 

The  next  question  put  students  into  the  position  of  thinking  how  they  would 
organize  a  DL  course:  ‘Please  respond  to  the  following  question.  If  I  were  building  an  on¬ 
line  course  to  teach  space  systems-related  instruction,  I  would...’  The  students  gave 
many  responses  to  this  question,  and  after  analysis  of  their  content  I  formed  six  different 
groups  of  responses  based  on  similar  themes  in  their  comments.  Here  are  the  grouped 
responses: 

Just  like  this  course 

•  ...  Do  exactly  what  you  are  doing. 

•  Copy  that  format! 

•  Do  exactly  what  you're  doing.  I  think  we're  getting  a  good  overview  of 
Space  and  are  taking  away  a  lot  of  useful  information. 

•  Model  it  pretty  much  like  you  have 

•  Not  change  a  damn  thing 

•  Try  to  do  as  good  as  you  have...  Really!  It  is  a  good  program  with  a 
layout  that  makes  navigation  easy. 

I’m  too  new  to  say 

•  Don’t  know 

•  lam  just  starting;  ask  me  at  the  end 

•  I  haven't  really  thought  about  it,  but  I  am  a  novice  at  on-line  courses,  so 
my  input  wouldn't  really  contribute  anything  substantial  at  this  time 

Hire  someone 

•  Find  experts  on  distance  learning  through  technology  (which  has  been 
done  here  already) 

•  I  would  hire  someone  to  do  it,  because  it's  beyond  me 
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First,  fix  what  you  can 


Conduct  thorough  system  testing  and  evaluation  and  keep  the  class  small 
till  all  the  bugs  are  worked  out 

Ensure  that  the  various  pieces  were  working  before  beginning.  This  is  not 
meant  as  sarcastically  as  it  sounds. 

Ensure  the  system  is  up  and  running  before  starting  a  class  with  students.  I 
would  require  a  ‘dry  run’  of  the  entire  course  and  all  features  completed 

Td  get  a  better  feel  for  how  it  is  going  to  work  before  putting  online. 
Possibly  hold  this  'online'  class  IN  class,  using  it  as  a  group  first.  Seems  to 
be  too  many  unknowns.  Also,  it  would  REALLY  benefit  us  to  be  able  to 
meet  from  time  to  time  to  assess  the  class  and  where  all  of  us  stand.  I 
understand  that  the  point  of  the  class  is  so  the  distance  learning  can  occur, 
but  since  the  class  is  in  its  genesis,  perhaps  meeting  once  every  two  weeks 
would  provide  insight  that  the  forum  cannot. 

Keep  fixing  the  bugs  and  add  more  user  help  key  like  spell  check  and 
summary  charts  for  graded  answers 

Work  out  the  bugs  ahead  of  time  - 1  think  you  have  done  a  great  job  with 
the  layout  and  interface  thus  far 

Content  improvements 

Add  CCTV/tape  like  instruction  by  instructor  on  key  points 

Arrange  online  chats  with  industry  specialists.  Follow  the  same  format  as 
presented  (but  without  the  bugs). 

Have  more  mathematical  or  practical  applications 

Provide  links  (outsource  or  internal)  that  would  provide  a  hands-on 
approach  to  solving  some  of  the  questions/problems,  i.e.,  orbital  model 
that  actually  used  the  information  that  we  input 


Instruction  methods  improvements 

Continue  to  require  the  Professor  to  keep  students  on  track  via  email. 
Those  emails  add  confidence  and  understanding  to  the  student  as  what  the 
instructor  exactly  expected  them  to  accomplish.  A  syllabus  is  nice,  but  as 
always  changes  occur.  In  an  online  course,  little  changes  create  confusion 
even  worse  that  in  regular  class.  Those  emails  from  the  professor  help. 

Ensure  that  complete  communication  via  email  is  always  available  letting 
me  know  when  forums  have  been  replied  to,  when  quizzes  and  evaluation 
are  opened 


•  Have  to  learn  a  lot  more  about  both  space  systems  and  distance  education 

•  Keep  the  learning  process  as  well  as  the  testing  setup  simple  and 
straightforward 

•  Limit  the  length  of  the  postings  and  smaller  group  forums  sooner 

•  One  thought.  In  the  forum  people  write  very  LARGE  paragraphs.  It  is 
difficult  to  read  online  and  follow.  Part  of  the  rules  of  the  road  should  be 
to  write  4/5  sentences  and  then  make  a  break  and  continue.  Just  like  I  am 
doing  now.  I  would  argue  people  would  be  more  inclined  to  read  5/6 
blocks  of  text  vise  one  humungous  block  of  text. 

•  Use  a  great  textbook  like  the  one  we  have  and  encourage  students  with 
regular  e-mails  (good  job!) 


The  last  question  in  this  questionnaire  asked  students  if  they  had  suggestions:  ‘Do 
you  have  any  suggestions,  at  this  point  (in  particular,  about  your  orientation  to  taking  a 
course  on-line)?’  Eighteen  of  them  did  not  respond  or  answered  ‘None’,  ‘Nothing  yet’,  or 
‘Nothing  at  this  time’.  Three  students  responded  that  they  have  answered  this  question 
previously.  Here  are  the  answers  of  the  remaining  students: 

Advice  and  ideas 

•  It  seems  like  the  bases  are  pretty  well  covered.  We  can  interact  online  and 
get  questions  answered  quickly. 

•  Just  some  more  QC1 1  on  the  links  and  quizzes 

•  Limit  forum  size  to  10  people  (Forum  teams  A,  B,  C,  D...  Gold,  Blue, 
Red,  Green...  Whatever...  Poring  through  a  plethora  of  responses  can  be 
somewhat  tiring... 

•  The  forum  discussions  were  lagging  during  the  technical  chapters 

Positive  opinions 

•  A  hesitant  ‘I’d  do  it  again  based  on  this  course’.  The  caveat  is  that  some 
courses  (like  Calculus)  wouldn't  lend  themselves  as  well  to  online 
teaching  (in  my  humble  opinion). 

•  Make  sure  requirements  of  the  students  are  clearly  defined  at  all  times 


1  *  Quality  control. 
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•  The  orientation  was  good  and  the  tech  support  has  been  very  responsive 

Neutral 

•  None  at  this  time.  I  think  it  is  a  good  idea  to  combine  the  quizzes  when 
technical  difficulties  are  encountered;  it  keeps  things  tracking  along  to  the 
next  module. 

Second  questionnaire  -  Summary 

Statements  on  comfort  levels  showed  that  positive  responses  dominate,  but  the 
distributions  of  the  student  responses  are  not  the  same  in  all  cases.  For  instance,  the 
highest  level  of  comfort  the  students  had  was  with  accessing  the  Internet.  Issues  like 
setting  up  one’s  own  system,  posting  to  forums,  or  WebCT  technical  support  had  some 
negative  responses  also.  Some  statements  clearly  had  the  highest  level  of  comfort,  but 
some  students  gave  negative  responses  anyway  as  in  the  statements  about  using  word 
processing  or  sending  e-mail. 

A  few  statements  produced  an  exceptionally  large  number  of  neutral  responses 
such  as  the  statement  on  ISP  technical  support  or  on  ADP  office  support.  This  occurred 
because  most  students  did  not  use  those  services. 

The  open-ended  question  about  needed  technical  help  clearly  indicated  that 
posting  to  forums  was  a  central  problem. 

Eight  statements  asked  the  students  about  the  comfort  level  with  the  use  of 
different  features  in  the  course:  forums,  e-mail,  quiz,  links,  course  overview,  module 
objectives,  rules  of  engagement,  and  course  syllabus.  Distributions  differ  among  these 
variables,  although  students  mostly  chose  positive  responses.  Sending  e-mail  had  the 
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most  responses  of  ‘extremely  comfortable’,  but  there  were  some  students  who  felt 
uncomfortable  even  with  that. 

Asked  about  additional  help  with  web-based  features,  six  students  emphasized 
problems  with  the  quiz,  but  there  were  other  responses  also. 

Another  open-ended  question  asked  the  students  what  they  liked  about  the  on-line 
course.  Nineteen  comments  could  be  classified  as  ‘time  management  flexibility  and 
accessibility’.  Other  features  were  mentioned  also:  forums  (six  times,  alone  or  in 
combinations),  links  (four  responses),  quizzes,  and  support. 

I  divided  the  responses  to  questions  about  student  concerns  into  the  following 
groups:  technical  problems  (four  responses);  uncertainty  about  comparability  of  DL  to 
face-to-face  classes  (three);  learning  pace  and  organization  (three);  grading  (three); 
combinations  and  other  (four).  Only  two  students  gave  very  positive  responses  to  this 
question. 

Near  the  end  of  the  questionnaire,  students  were  asked  to  respond  to  the  following 
statement:  ‘If  I  were  building  an  on-line  course  to  teach  space  systems-related  instruction, 
I  would...’  Their  responses  were  grouped  around:  ‘Instructional  methods  improvements’ 
(seven  responses);  ‘I  would  do  just  same  as  in  this  course  (six  responses);  ‘First,  fix  all 
what  you  can’  (six);  ‘I’m  too  new  to  say’  (three);  and  ‘Hire  someone’  (two  responses’. 

3.  Module  14  Questionnaire 

This  questionnaire  (Appendix  C)  was  administered  immediately  after  module  14 
was  completed,  i.e.,  close  to  the  end  of  the  course  (the  course  had  15  modules).  Since 
module  14  was  different  from  the  others  (special  guest  and  a  subject  matter  expert  was 
available  for  interactions  with  the  students),  the  instructor  wanted  to  learn  how  much  the 
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students  benefited  from  the  module  14  learning  objectives.  The  instructor  formulated 
these  statements. 

I  added  several  questions  about  the  current  comfort  level  with  important  features 
used  in  the  course  so  that  the  changes  in  comfort  levels  from  the  beginning  of  the  course 
to  date  could  be  examined. 

Since  the  students  for  this  on-line  questionnaire  were  identifiable  by  name,  it  was 
possible  to  compare  their  responses  to  this  questionnaire  with  data  from  other 
instruments:  the  first  questionnaire,  student  tracking  data,  and  even  final  course  grades. 

First  the  students  were  asked  to  respond  on  seven-point  scales  to  seven  statements 
about  achieving  learning  objectives  in  Module  14.  The  next  six  statements  provided  were 
about  the  student  comfort  level  with  features  used  in  the  course.  At  the  end  of  this 
questionnaire,  the  students  were  given  an  opportunity  to  add  their  own  comments. 

Twenty-six  students  completed  this  questionnaire. 

The  first  question  was:  ‘Throughout  Module  14  you  developed  and  practiced  your 
skills  in  the  process  of  analyzing,  articulating,  and  evaluating  DoD  Space  Control  issues. 
The  statements  below  reflect  the  learning  objectives  for  Module  14.  To  what  extent  do 
you  agree  that  the  material  presented  in  the  Module  enabled  you  to  achieve  the  learning 
objectives?  Identifying  key  elements  of  Space  Control.’  Figure  27  shows  the  distribution 
of  the  responses. 
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"identifying  Key  Elements  of  Space  Control" 


Number  of 
students 


4* 


Figure  27.  Distribution  of  the  Responses  to  the  First  Statement. 

N  =  26  Mean  =  5.92  Median  =  6 

Minimum  =  2  Maximum  =  7  Mode  =  6 


Nineteen  students  (73.08%)  responded  with  ‘agree’,  so  this  learning  objective  was 
very  successfully  met  with  the  module.  This  variable  has  significant  correlations  with 
fourteen  variables  from  this  and  other  instruments  (Table  28). 


Instrument 

Statement  /  Variable 

P 

WWTTfll 

Module  14 

4  -  ‘Participating  in  forums  (e.g.  discussing, 
identifying/consulting  subject  matter  experts),  to 
evaluate  space  control  issues.’ 

0.559 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.494 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.470 

0.015 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

0.440 

0.024 

Fist 

questionnaire 

‘I  expect  this  online  course  will  be  valuable  and 
improve  my  learning’ 

0.405 

0.026 

Student 

tracking 

Number  of  follow-up  posts 

0.367 

0.046 

Table  28.  Significant  Correlations  and  Levels  of  Significance  between  the  First 

Statement  and  other  Variables  (Ordered  by  Level  of  Significance). 
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Correlations  in  Table  28  show  that  the  students  with  a  higher  comfort  level  in  sending  e- 
mail,  accessing  the  Internet,  and  the  quiz  feature,  said  that  the  course  material  helped 
them  in  higher  achievement  in  the  first  learning  objective.  Also,  they  tended  to  have 
somewhat  higher  expectations  to  improve  their  learning  throughout  the  course  and  to 
send  more  follow-up  posts  in  forums. 

The  second  question  was:  ‘Synthesizing  other’s  findings  in  Forums’  (see 
explanation  with  first  question).  Figure  28  shows  the  distribution  of  the  responses. 


Figure  28.  Distribution  of  the  Responses  to  the  Second  Statement. 

N  =  26  Mean  =  5.19  Median  =  5 

Minimum  =  5  Maximum  =  7  Mode  =  5 


Predominant  responses  to  the  second  statement  were  ‘slightly  agree’  and  ‘agree’ 

(ten  students  or  38.46%  chose  each).  Responses  did  not  have  any  significant  correlations 

with  other  variables,  which  would  suggest  that  this  learning  objective  is  specific,  without 

any  apparent  relation  to  any  other  of  the  variables  measured  with  this  set  of  instruments. 
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The  third  statement  in  the  questionnaire  was:  ‘Interacting  with  a  subject-matter 
expert  whose  knowledge  you  can  use  to  solve  space-system  problems.’  Figure  24  shows 
the  distribution  of  the  student  responses. 


"Interacting  with  a  Subject  Matter  Expert." 


Figure  29.  Distribution  of  the  Responses  to  the  Third  Statement. 

N  =  26  Mean  =  4.85  Median  =  5 

Minimum  =  1  Maximum  =  7  Mode  =  4 

Here  the  student  responses  vary  more  than  the  responses  to  the  first  two 

statements.  Seven  students  (26.93%)  chose  neutral  answers,  which  might  be  an  indicator 

that  they  personally  have  not  taken  the  opportunity  to  enter  into  interactions  with  guest  - 

subject  matter  expert,  or  they  saw  this  module  in  terms  of  the  stated  learning  objectives 

exactly  ‘between  successful  and  unsuccessful’.  This  variable  had  significant  correlations 

with  four  other  variables  (Table  29). 

Students  with  higher  expectations  in  improving  their  learning  in  this  course  saw 

this  interaction  as  more  useful  for  their  learning.  Also,  they  tended  to  value  participation 
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in  forums  more,  and  they  also  tended  to  participate  more  in  forums  by  posting  messages 
and  follow-up  posts  to  forums. 


Instrument 

Statement  /  Variable 

P 

p  value 

Fist 

questionnaire 

‘I  expect  this  online  course  will  be  valuable  and 
improve  my  learning’ 

0.450 

0.012 

Module  14 

4  -  ‘Participating  in  forums  (e.g.  discussing, 
identifying/consulting  subject  matter  experts),  to 
evaluate  space  control  issues.’ 

0.439 

0.025 

Student 

tracking 

Number  of  posted  messages 

0.404 

0.027 

Student 

tracking 

Number  of  follow-up  posts 

0.414 

0.023 

Table  29.  Significant  Correlations  and  Levels  of  Significance  between  the  Third 
Statement  and  other  Variables  (Ordered  by  Level  of  Significance). 

The  fourth  statement  was  the  following:  ‘Participating  in  forums  (e.g.  discussing, 
identifying/consulting  subject  matter  experts),  to  evaluate  space  control  issues.’  Figure  30 
shows  how  the  student  responses  were  distributed.  Eighteen  students  agreed  (69.23%)  - 
six  ‘somewhat  agreed’  (23.08%),  nine  ‘agreed’  (34.61%),  and  three  ‘extremely  agreed’ 
(11.54%),  but  there  were  five  neutral  (19.23%)  and  three  responses  (11.54%)  that 
disagreed  with  the  statement. 

Table  30  shows  four  significant  correlations  this  variable  had  with  other  variables. 
Insight  into  the  variables  in  the  table  reconfirms  the  relationship  between  valuing 
interactivity,  valuing  student  participation  (in  forums),  and  higher  expectations  in 
learning  throughout  the  course.  A  negative  sign  of  correlation  between  the  responses  to 
the  fourth  statement  and  the  question  from  the  first  questionnaire  about  concerns  of  one’s 
own  abilities  means  that  the  students  less  concerned  about  their  own  abilities  to  use  web- 
based  materials  in  the  on-line  class  tend  to  see  material  in  module  14  as  helping  them 
more  in  participating  in  forums  and  evaluating  space  control  issues. 
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"Participating  in  Forums,  to  Evaluate  Space  Control 

Issues" 


9 

8 

7 

6 

Number  of  5 

students  4 
3 

2 

1 

0 


-<©e  (tf©c  A<ee  rfee  A<ee  A^ee 

SS®9  ^  *«/ 

S'-'0 


Figure  30.  Distribution  of  the  Responses  to  the  Fourth  Statement. 


N  =  26  Mean  =  5.12  Median  =  5 

Minimum  =  2  Maximum  =  7  Mode  =  6 


Instrument 

Statement  /  Variable 

P 

p  value 

Module  14 

1  -  ‘Identifying  key  elements  of  Space  Control.’ 

0.559 

Module  14 

3  -  ‘Interacting  with  a  subject-matter  expert  whose 
knowledge  you  can  use  to  solve  space-system 
problems.’ 

0.439 

0.025 

Fist 

questionnaire 

‘I  expect  this  online  course  will  be  valuable  and 
improve  my  learning’ 

0.392 

0.032 

First 

questionnaire 

‘I  am  concerned  about  my  ability  to  use  the  web- 
based  materials  in  this  class’ 

-0.364 

0.048 

Table  30.  Significant  Correlations  and  Levels  of  Significance  between  the  Fourth 
Statement  and  other  Variables  (Ordered  by  Level  of  Significance). 

All  the  students  agreed  with  the  fifth  statement,  except  two  who  gave  neutral 
responses.  Materials  in  module  14  successfully  helped  them  in  learning  this  learning 
objective  (Figure  31). 
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"Articulating  How  the  Concepts  of  Space  Control 
Relate  to  Joint  Vision  2020" 


Figure  3 1 .  Distribution  of  the  Responses  to  the  Fifth  Statement. 

N  =  26  Mean  =  5.85  Median  =  6 

Minimum  =  4  Maximum  =  7  Mode  =  6 

Four  correlations  were  significant,  two  on  other  learning  objectives  in  this 
module,  and  two  on  the  comfort  level  with  accessing  the  Internet  and  with  sending  e-mail 
(Table  31).  Interpretation  is  not  easy,  but  students  with  higher  comfort  levels  with  basic 
technical  tools  in  the  DL  course  should  be  expected  to  be  more  successful  in  this  kind  of 
learning  environment. 

The  sixth  statement  was  ‘Evaluating  the  likelihood  that  various  types  of  space 
countermeasures  might  be  used  across  the  peace-war  continuum’.  Responses  were 
favorable,  without  ‘disagree’  responses  except  only  one  student.  Figure  32  shows  the 
distribution  of  the  responses.  Similarly  to  other  statements  in  this  set,  there  are  few 
significant  correlations  -  two  with  variables  from  this  same  set  of  statements,  one  with 


99 


variable  from  the  first  questionnaire,  and  one  with  the  number  of  visits  to  the  course 
homepage  (Table  32). 


Instrument 

Statement  /  Variable 

P 

p  value 

Module  14 

6  -  ‘Evaluating  the  likelihood  that  various  types  of 
space  countermeasures  might  be  used  across  the 
peace- war  continuum.’ 

0.525 

Module  14 

7  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.448 

0.022 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.416 

0.035 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.388 

0.050 

Table  3 1 .  Significant  Correlations  and  Levels  of  Significance  between  the  Fifth 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 


"Evaluating  the  Likelihood  that  Space 
Countermeasures  Might  be  Used" 


Figure  32.  Distribution  of  the  Responses  to  the  Sixth  Statement. 


N  =  26 


Mean  =  5.96  Median  =  6 


Minimum  =  4 


Maximum  =  7  Mode  =  6 
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Instrument 

Statement  /  Variable 

P 

p  value 

Module  14 

7  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.579 

0.002 

Module  14 

5  -  ‘Articulating  how  the  concept  of  space  control 
relates  to  Joint  Vision  2020.’ 

Student 

tracking 

Number  of  visits  to  course  homepage 

0.408 

0.025 

Fist 

questionnaire 

‘How  would  you  rate  your  on-line  search  resources 
and  techniques?’ 

0.367 

0.046 

Table  32.  Significant  Correlations  and  Levels  of  Significance  between  Sixth 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  last  statement  in  this  set  was:  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction  by  adversaries.’  Figure  33  shows  the 
distribution  of  the  responses,  which  are  in  this  case  also  very  favorable  with  only  one 
student  disagreeing  slightly.  Significant  are  the  correlations  with  two  other  statements 
from  this  set:  the  self-perceived  on-line  search  skills  from  first  questionnaire  and  the 
comfort  level  with  the  links  feature  (Table  33). 


"Identifying  Elements  of  Space  Systems  Vulnerabile 
to  Disruption,  Degradation,  or  Destruction..." 
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Figure  33.  Distribution  of  the  Responses  to  the  Seventh  Statement. 
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N  =  26 
Minimum  =  3 


Mean  =  6.23 
Maximum  =  7 


Median  =  6 
Mode  =  6 


Instrument 

Statement  /  Variable 

. P 

Module  14 

6  -  ‘Evaluating  the  likelihood  that  various  types  of 
space  countermeasures  might  be  used  across  the 
peace- war  continuum.’ 

0.579 

Module  14 

5  -  ‘Articulating  how  the  concept  of  space  control 
relates  to  Joint  Vision  2020.’ 

0.022 

Fist 

questionnaire 

‘How  would  you  rate  your  on-line  search  resources 
and  techniques?’ 

0.398 

0.030 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

0.398 

0.044 

Table  33.  Significant  Correlations  and  Levels  of  Significance  between  the  Seventh 

Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  first  set  of  statements  was  followed  by  six  questions  on  the  comfort  level  with 
different  features  used  in  the  course.  These  questions  are  similar  or  identical  to  the 
question  used  in  Questionnaire  A-2  so  that  comparisons  are  possible.  The  eighth  item  in 
the  questionnaire  asked  students  about  their  comfort  level  with  accessing  the  Internet 
(Figure  34). 


102 


N  =  26  Mean  =  6.77  Median  =  7 

Minimum  =  6  Maximum  =  7  Mode  =  7 

This  distribution  is  very  similar  to  responses  to  the  identical  question  in  the  A-2 
questionnaire  (Figure  11).  As  in  responses  to  the  same  question  in  A-2,  significant 
correlations  are  high  and  numerous  (Table  34,  compare  to  Table  13).  On  both  occasions, 
when  this  question  was  asked,  the  highest  correlations  were  with  the  comfort  level  with 
links,  use  of  e-mail,  and  postings  to  forums.  In  this  questionnaire  there  are  some  variables 
not  present  in  the  data  analysis  in  the  A-2  questionnaire  such  as  the  set  of  questions  about 
features  used  in  the  course  (rules  of  engagement,  module  objectives,  and  on-line  student 
tracking  data).  Some  of  the  variables  from  these  sets  also  appeared  to  be  statistically 
significant,  including  three  indicators  from  student  tracking. 
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Instrument 

Statement  /  Variable 

P 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

0.794 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.778 

<  0.001 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

0.695 

<0.001 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

0.656 

<0.001 

Module  14 

13  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.653 

<0.001 

Student 

tracking 

Number  of  visits  to  homepage 

0.475 

0.005 

Student 

tracking 

Number  of  articles  read 

0.461 

0.010 

Module  14 

1  -  ‘Identifying  key  elements  of  Space  Control.’ 

0.470 

Student 

tracking 

Total  number  of  hits 

0.425 

0.019 

Module  14 

5  -  ‘Articulating  how  the  concept  of  space  control 
relates  to  Joint  Vision  2020.’ 

0.035 

Table  34.  Significant  Correlations  and  Levels  of  Significance  between  the  Eighth 
Statement  and  other  Variables  (Ordered  by  Level  of  Significance). 

The  next  question  was  about  the  comfort  level  with  sending  e-mail  (Figure  35). 
Now  all  the  students  felt  comfortable  or  extremely  comfortable  sending  e-mail,  which 
was  not  the  case  in  the  A-2  questionnaire  (Figure  14),  where  three  students  expressed 
they  felt  somewhat  (one  student)  or  even  extremely  uncomfortable  (two  students). 
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Comfort  Level  with  Sending  E-mail 


Figure  35.  Distribution  of  the  Responses  to  the  Ninth  Statement. 

N  =  26  Mean  =  6.85  Median  =  7 

Minimum  =  6  Maximum  —  7  Mode  =  7 

The  dominant  issues  with  the  highest  correlations  seem  to  be  different  than  in  the 
A-2  questionnaire  (Table  35  below,  compare  with  Table  15).  In  the  latter  case,  the 
highest  correlation  was  with  the  comfort  level  with  word  processing,  which  is  not 
significant  in  this  questionnaire.  There  are  also  some  other  variables  that  are  statistically 
significant  such  as  four  correlations  with  on-line  tracking  indicators.  Correlations  with 
the  on-line  tracking  data  indicate  that  students  more  comfortable  with  sending  e-mail  tend 
to  read  more  articles,  visit  the  course  homepage  more  often,  post  more  articles,  and  in 
general  visit  on-line  course  material  more  frequently.  Overall,  comfort  level  with  sending 
e-mail  seems  to  be  important  for  student  achievement  in  the  course. 
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The  next  question  was:  ‘How  would  you  rate  your  comfort  level  at  this  point  with 
posting  comments  to  the  Forums?’  Figure  36  shows  the  distribution  of  the  responses. 
Twelve  students  (46.15%)  felt  ‘extremely  comfortable’  with  posting  to  the  forums,  two 
students  (7.69%)  felt  ‘somewhat  comfortable’,  and  two  even  ‘somewhat  uncomfortable’. 
Compared  to  the  distribution  of  the  responses  to  a  similar  question  in  A-2  questionnaire 
(Figure  13),  this  distribution  shows  that  the  comfort  level  improved  during  the  course. 


Instrument 

Statement  /  Variable 

P 

p  value 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.778 

<0.001 

Module  14 

13  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.662 

<0.001 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

0.600 

0.001 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

0.567 

0.003 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

Module  14 

1  -  ‘Identifying  key  elements  of  Space  Control.’ 

0.494 

0.010 

Student 

tracking 

Number  of  articles  read 

0.420 

0.021 

Student 

tracking 

Number  of  visits  to  homepage 

0.400 

0.029 

Student 

tracking 

Number  or  articles  posted 

Student 

tracking 

Total  number  of  hits 

0.387 

0.034 

Module  14 

5  -  ‘Articulating  how  the  concept  of  space  control  - 
relates  to  Joint  Vision  2020.’ 

0.388 

0.050 

Table  35.  Significant  Correlations  and  Levels  of  Significance  between  the  Ninth 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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Similar  items  appear  on  the  list  of  highly  correlated  variables,  and  correlations 
with  five  student  tracking  indicators  are  statistically  significant  also  (Table  36  below, 
compare  with  Tables  14  and  19). 


Comfort  Level  with  Posting  Comments  to  the 
Forums 


Figure  36.  Distribution  of  the  Responses  to  the  Tenth  Statement. 

N  =  26  Mean  =  6.12  Median  =  6 

Minimum  =  3  Maximum  =  7  Mode  =  7 

A  question  on  the  comfort  level  with  the  quiz  feature  was  next.  The  distribution  of 
the  responses  (Figure  37)  had  much  more  responses  about  high  comfort  level  than  in  the 
questionnaire  A-2  (Figure  21).  However,  this  change  is  not  only  due  to  the  learning 
process  and  improving  one’s  own  skills  in  the  course,  but  also  due  to  the  fact  that 
technical  problems  such  as  ‘bugs’  in  software  were  fixed  after  the  A-2  was  administered. 

This  variable  has  numerous  significant  correlations  with  other  variables  (Table 
37),  including  six  student  tracking  indicators  and  final  grades.  The  instructor  did  not 
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include  quiz  scores  in  the  final  grade,  but  the  correlation,  even  though  not  high12,  is 
evident.  Also,  there  are  similarities  to  correlations  in  the  A-2  question  about  the  quiz 
feature.  Compare  it  to  Table  23. 


Instrument 

Statement  /  Variable 

P 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

0.687 

<  0.001 

Module  14 

13  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.669 

<  0.001 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.656 

<  0.001 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.567 

0.003 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

0.565 

0.003 

Student 

tracking 

Number  of  original  posts 

0.485 

0.007 

Student 

tracking 

Total  number  of  hits 

0.441 

0.015 

Student 

tracking 

Number  or  articles  posted 

! 

0.416 

0.022 

Student 

tracking 

Number  of  visits  to  homepage 

0.406 

0.026 

Student 

tracking 

Number  of  articles  read 

0.373 

0.043 

Table  36.  Significant  Correlations  and  Levels  of  Significance  between  the  Tenth 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 


12  By  ‘high’  correlation  statisticians  usually  mean  values  over  0.75  or  0.80. 
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Comfort  Level  with  Quiz  Feature  Used  in  the  Course 
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Figure  37.  Distribution  of  the  Responses  to  the  Eleventh  Statement. 


N  =  26  Mean  =  6.35  Median  =  7 

Minimum  =  3  Maximum  =  7  Mode  =  7 

The  question  about  the  links  feature  has  the  highest  percent  of  ‘extremely 
comfortable’  responses  -  76.92%  (Figure  38).  Only  one  student  felt  ‘somewhat 
uncomfortable’  with  the  links  used  in  the  course.  Similar  responses  about  this  feature 
were  in  questionnaire  A-2  (Figure  17),  but  with  one  ‘extremely  uncomfortable’  response. 
There  is  only  one  very  high  correlation  (Table  38)  with  the  comfort  level  in  accessing  the 
Internet.  This  is  logical  because  problems  with  access  certainly  cause  many  other 
problems  in  using  the  course  features,  which  includes  links,  but  the  other  five  correlations 
were  statistically  significant  also.  Interestingly,  the  number  of  significant  correlations  was 
higher  in  A-2  questionnaire  (Table  22). 
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Instrument 

Statement  /  Variable 

P 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.695 

<0.001 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.600 

0.001 

Student 

tracking 

Total  number  of  hits 

0.581 

0.001 

Module  14 

13  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.542 

0.004 

Student 

tracking 

Number  of  articles  posted 

0.497 

0.005 

Student 

tracking 

Number  of  articles  read 

Student 

tracking 

Number  of  visits  to  homepage 

0.449 

0.013 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

0.475 

FINAL  GRADE 

0.465 

Student 

tracking 

Number  of  original  posts 

0.427 

0.019 

Module  14 

1  -  ‘Identifying  key  elements  of  Space  Control.’ 

0.440 

0.024 

Student 

tracking 

Number  of  follow-up  posts 

0.411 

0.024 

Table  37.  Significant  Correlations  and  Levels  of  Significance  between  the  Eleventh 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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Figure  38.  Distribution  of  the  Responses  to  the  Twelfth  Statement. 

N  =  26  Mean  -  6.62  Median  =  7 

Minimum  =  3  Maximum  =  7  Mode  =  7 

The  students  responded  to  the  thirteenth  question  about  the  comfort  level  with 
word  processing  in  a  similar  manner  as  the  comfort  level  with  links  used  in  the  course. 
The  vast  majority  (twenty  students  or  76.92%)  felt  extremely  comfortable,  which  is  what 
is  expected  from  students  at  NPS  (Figure  39).  This  distribution  shows  slightly  higher 
student  comfort  level  than  results  in  the  second  questionnaire  (compare  Figure  7).  Again, 
there  were  many  significant  correlations  with  other  variables  such  as  those  asking  about 
comfort  level  with  forums,  finals  grades,  e-mail,  accessing  the  Internet  etc.,  and  with  five 
student  tracking  indicators  (Table  39).  The  comfort  level  with  word  processing  seems  to 
be  very  indicative  of  students’  ‘broad  technical  proficiency’,  which  is  so  important  in  a 
DL  environment.  A  similarity  about  correlations  between  such  variables  is  present  in  the 
A-2  questionnaire  (Table  14). 
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Instrument 

Statement 

P 

p  value 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.794 

<  0.001 

Module  14 

1 0  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

0.565 

0.003 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.531 

0.005 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  word  processing?’ 

0.445 

0.023 

Module  14 

7  -  ‘Identifying  elements  of  space  systems  that  are 
vulnerable  to  disruption,  degradation  or  destruction 
by  adversaries.’ 

0.398 

0.044 

Table  38.  Significant  Correlations  and  Levels  of  Significance  between  the  Twelfth 
Statement  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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Figure  39.  Distribution  of  the  Responses  to  the  Thirteenth  Statement. 


N  =  26  Median  =  7  Minimum  =  5 

Maximum  =  7  Mode  =  7 
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Instrument 

Statement  /  Variable 

P 

p  value 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

0.669 

<0.001 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

0.662 

<0.001 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

0.653 

<0.001 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  quiz  feature  being  used  to  teach  this 
course?’ 

0.542 

0.004 

Student 

tracking 

Number  or  articles  posted 

0.501 

0.005 

Student 

tracking 

Number  of  articles  read 

0.478 

0.008 

FINAL  GRADE 

0.441 

0.015 

Module  14 

12-  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  the  links  (i.e.,  additional  web  sites) 
feature  being  used  to  teach  this  course?’ 

0.445 

0.023 

Student 

tracking 

Number  of  original  posts 

0.412 

0.024 

Student 

tracking 

Number  of  follow-up  posts 

0.407 

0.026 

Student 

tracking 

Total  number  of  hits 

0.404 

0.027 

Table  39.  Significant  Correlations  and  Levels  of  Significance  between  the  Thirteenth 
Statement  and  other  Variables  (Ordered  by  Level  of  Significance). 


The  last  question  in  the  Module  14  questionnaire  was  open-ended,  thereby  giving 
students  an  opportunity  to  add  comments.  The  comments  varied  from  simple  and  positive 
to  rather  complex.  The  students  also  gave  suggestions  for  course  improvements.  Here  are 
the  comments  grouped  into  five  types: 

Positive  comments 

•  This  course  is  a  great  for  keeping  up  on  current  events,  as  well  as  keeping 
our  interest  in  the  topic  by  linking  us  to  good  web  resources 

•  I  enjoyed  the  course!  I  look  forward  to  seeing  more  courses.  I  feel  this  is 
the  beginning  of  some  great  opportunities  for  troops  who  would  not 
otherwise  be  able  to  take  classes  due  to  location  or  OPTEMPO. 
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This  was  my  first  online  course  and  I  truly  enjoyed  it.  On-line  courses  take 
a  great  deal  of  personal  commitment  and  focus.  Providing  an  interesting 
subject  and  stimulating  discussion  helps  to  increase  participation. 

Having  no  prior  courses  in  space  systems,  I  did  not  know  what  to  expect.  I 
am  amazed  at  how  much  our  military  (as  well  as  everyday  life)  is 
dependent  on  space  systems.  This  especially  comes  as  a  surprise  having 
been  on  4  WESTPAC  deployments  and  not  being  aware  of  what's  going 
on  "up  there".  It  all  seemed  so  transparent...  not  anymore.  I  appreciate  this 
great  course,  which  will  ensure  I  will  continue  to  educate  myself  about 
space  systems. 

Honestly,  if  I  was  a  space  guy,  I  might  not  have  gotten  a  great  deal  for  this 
course  and  wouldn’t  have  been  challenged.  As  it  is,  I’m  an  ITM  guy  and 
feel  that  I  received  a  great  overview  of  Space,  the  systems  involved,  how 
the  systems  relate  to  the  military,  etc.  I’ll  admit  it;  I  actually  enjoyed  the 
course  and  looked  forward  to  reading  the  articles  and  postings.  This 
definitely  would  not  have  been  the  case  if  I’d  been  forced  into  learning  a 
lot  of  math  that  I  would  never  use  again  (and  would  thus  drop  out  of  my 
brain's  RAM  as  soon  as  the  quarter  was  over). 

As  I  stated  in  my  written  SOF,  had  testing  occurred  weekly,  it  probably 
would  have  kept  me  in  the  books/readings  with  more  consistency.  The 
readings  were  very  interesting... much  more  so  than  the  text....  Some  of  the 
text  concepts  (orbits)  were  difficult  to  understand  without  direct  classroom 
instruction...  Overall,  a  good  experience. 


Positive  comments,  mentioning  some  problems 

Although  I  enjoyed  the  flexibility  of  the  on-line  course,  I  am  frustrated 
with  the  content.  There  were  no  technical  discussions  or  material  provided 
in  the  forums  or  quizzes.  The  required  understanding  of  the  material  was 
rudimentary  at  best.  If  this  was  meant  as  an  introductory  course,  then  it 
met  the  requirement.  If  it  was  meant  as  a  technical  course,  then  it  fell  way 
short. 

This  course  is  a  good  example  of  what  you  can  do  in  an  Internet  course.  I 
thought  the  forum  played  a  good  role  in  student  interaction.  The  loss  of  the 
ability  to  quiz  on  later  sections  did  tend  to  demotivate.  I  know  it  should  be 
more  than  grades  and  testing,  but  with  our  schedules,  you  do  what  you  are 
tested  on.  All  around  the  course  was  a  good  experience. 

Outstanding  links  were  provided  to  support  objectives  of  each  module. 
Very  thoughtful,  innovative,  and  interesting  presentations.  The  class 
seemed  to  get  a  little  off-track  when  the  system  kept  going  down,  but 
made  a  strong  finish.  System  reliability  is  a  problem  that  needs  to  be 
addressed.  Perhaps  a  backup  server  at  a  different  location.  I  really  enjoyed 
this  class. 


•  I  think  there  is  still  plenty  of  room  for  improvement.  Network  downtime 
and  glitches  interrupted  the  flow  of  the  course.  I  believe  the  Quizzes  for 
each  module  are  a  necessity.  While  they  are  a  pain  in  the  rear,  they  really 
force  you  to  keep  up  with  the  material...once  the  quizzes  tapered  off. . .  It  is 
easy  to  get  behind.  The  quizzes  don't  have  to  be  big...just  enough  to  keep 
you  honest.  Other  than  that  I  think  the  course  has  great  potential  for 
learning. 


Some  ideas  and  suggestions 

•  Would  like  to  see  a  “What’s  New”  page  every  time  I  log  on  some  can 
instantly  see  what  is  happening  relative  to  bug  fixes,  or  new 
enhancements,  etc.  Also  would  like  an  email  feature  that  would  let  me 
know  when  someone  replied  to  a  forum  posting  of  my  own. 

•  I  think  there  should  be  quizzes  after  each  module  to  keep  us  on  our  toes! 
Aside  from  that,  it  was  a  very  interesting  course. 

•  I  think  that  forums  every  week,  in  small  groups  that  change,  would  have 
drastically  increased  my  consistent  participation  in  the  course.  I  would 
even  say  that  part  of  the  individual  grade  for  a  student  would  be  made  up 
of  the  average  grade  of  all  the  groups  he  or  she  belonged  to.  Nothing  like  a 
little  peer-pressure  to  get  a  group  to  encourage  itself  (member  to  member 
e-mails  or  whatever). 


Skeptics 

•  An  on-line  course,  will  never  provide  the  same  quality  of  instruction  as  an 
actual  in  class  course.  I  do  not  feel  like  I  got  more  than  a  cursory 
knowledge  of  the  material  covered.  The  on-line  format  is  adequate  for 
training  type  courses-  not  education.  To  improve  the  class,  there  should 
be  at  least  one  class  meeting  per  week  (with  VTC  if  required).  All 
modules  should  be  available  at  all  times  in  order  to  work  at  your  own  pace 
and  get  ahead  when  you  can.  In  the  future,  I  will  be  much  more  less 
inclined  to  take  an  on-line  class.  If  this  is  the  wave  of  the  future,  our 
educational  system  is  in  bigger  trouble  than  we  think, 

•  I  felt  I  would  have  gotten  more  from  a  traditional  course.  I  understand 
there  were  some  technical  problems  with  the  course  but  I  believe  the  face- 
to-face  contact,  at  lease  occasionally  is  really  necessary. 


Complex  comments 

The  course  was  difficult  to  complete  online.  Classroom  interface  was 
needed.  I  think  may  be  adding  a  Chat  Room  and  class  meeting  time  would 
help  get  a  better  understanding  of  what  was  required.  The  emails  were 
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great  help  to  keep  me  on  track.  However,  the  emails  were  late  in  support 
of  the  modules.  The  modules  were  too  dynamic  for  me  to  follow.  I  think 
the  webs  site  should  put  old  modules  or  incorrect  data  on  the  web  site. 
Most  of  the  Quizzes  didn't  work  correctly.  My  quiz  grade  is  still  messed 
up.  The  interface  with  a  keyboard  is  slow  and  painful.  Spell  check  and 
grammar  check  would  help.  I  did  learn  a  lot  about  space  systems  from  the 
links.  At  this  point  I  would  not  hire  a  person  with  a  degree  earn  online. 
The  student  work  is  hard  to  verify.  Think  of  adding  a  private  chat  room  for 
instructor  to  student  test.  Online  Questions  could  be  asked  in  a  private 
chat  as  a  test  question. 

•  I  would  have  enjoyed  this  class  more  if  it  were  my  only  class.  It  was  a 
serious  time  HOG  when  it  came  to  posting  messages.  What  you  could  do 
in  2  minutes  discussing  all  the  point,  it  would  take  45  minutes  to  an  hour 
to  write  a  well-articulated  point  paper...  I  think  messages  should  be  limited 
to  7/8  sentences  max.  Some  people  wrote  books,  which  subconsciously 
made  the  stakes  higher  for  everyone  in  trying  to  write  something  unique. 
Format  of  message  post... to  aid  reading,  users  must  write  3/4  sentences 
and  put  a  break  and  start  on  the  next  line.  This  should  be  in  the  rules  of 
engagement.  Too  many  posts  ran  forever  all  together,  and  made  it  difficult 
to  read.  The  whole  forum  message  board  is  not  extremely  intuitive.  I  have 
it  figured  out  but  there  are  a  lot  of  menu  items.  Unsure  as  to  you  could  do 
to  improve.  Personally,  I  liked  the  online  quizzes.  Learning  does  take 
place  when  taking  them.  Bugs  need  to  be  worked  out.  Frustrating  when  the 
quizzes  did  not  score  correctly  initially. 

•  Strong  points: 

o  The  information  provided  on  the  web  site  was  excellent.  The  links 
were  informative  and  rich  with  both  interesting  and  relevant 
information.  Most  definitely,  the  best  part  of  the  course, 
o  The  forums  allowed  for  an  exchange  of  ideas.  It's  a  good  way  to 
learn. 

•  Weak  points: 

o  I  could  have  done  without  the  textbook.  The  website  provided  all 
the  book  had  and  more.  I  really  don't  think  there’s  a  need  for  a 
textbook. 

o  The  forums  weren't  really  a  hotbed  of  new  ideas  or  even  an 
exploration  of  old  ideas.  In  the  future,  some  method  of  more 
regular  interaction  should  be  developed. 

•  Overall: 

o  It  is  very  obvious  that  a  great  deal  of  work  and  time  went  into 
developing  this  course.  I  think  many  of  the  bugs  experienced 
during  this  quarter  for  this  course  were  temporary.  This  will  be  one 
heck  of  a  course  (on-line  or  otherwise)  when  everything  is  up  and 
running? 

O 
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Module  14  Questionnaire  -  Summary 

This  on-line  questionnaire  was  administered  after  module  14.  In  total  15  modules 
were  covered  in  the  course.  This  questionnaire  was  not  anonymous;  twenty-six  students 
completed  it. 

There  were  three  groups  of  questions/statements.  The  first  seven  were  designed 
by  the  instructor  to  see  how  well  the  module  14  learning  objectives  were  being  met.  The 
next  six  questions  asked  the  students  about  their  comfort  level  with  relevant  ‘technical’ 
issues  and  features  used  in  the  course  such  as  accessing  the  Internet,  sending  e-mail, 
posting  comments  to  forums,  completing  the  quiz  feature  used  in  the  course,  using  the 
links  feature  and  lastly,  using  word  processing.  The  last  question  was  open-ended; 
students  had  an  opportunity  to  add  any  comments  or  additional  thoughts. 

Responses  to  the  first  seven  statements  were  predominantly  positive,  but  with 
slightly  different  distributions.  Some  statements  had  a  very  high  number  of  ‘agree’  and 
‘strongly  agree’  responses,  and  some  others  had  a  significant  number  of  ‘disagree’ 
responses.  In  general,  the  students  responded  that  all  module  objectives  were  more  or  less 
met  to  a  very  satisfactory  degree. 

Responses  about  comfort  levels  with  features  mentioned  above  are  very  similar  to 
the  responses  to  the  corresponding  questions  in  the  Second  questionnaire  (A-2),  but  with 
increased  overall  comfort  level.  Student  on-line  tracking  indicators  and  some  items  from 
the  First  questionnaire  had  significant  correlations  with  several  questions  in  this 
questionnaire,  like  questions  about  comfort  level  with  accessing  the  Internet,  with 
sending  e-mail,  with  posting  to  forums,  etc. 
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The  student  comments  provided  many  interesting  comments,  which  were  grouped 
into  the  following  content  types:  positive  comments  (six  students);  positive  comments 
with  mentioning  some  problems  (four);  ideas  and  suggestions  (three);  complex  comments 
(three);  and  ‘skeptics’  (two  students).  Seven  students  did  not  add  any  comments. 

4.  Final  Survey 

This  on-line  questionnaire  was  administered  after  the  course  was  completed,  after 
the  final  exam.  However,  it  was  administered  before  the  she  instructor  gave  final  grades 
to  the  students.  It  was  anonymous.  Twenty-three  students  completed  it. 

The  thirty-one  items  included: 

•  Questions  reflecting  course  objectives 

•  Two  questions  on  student  self-confidence  in  certain  activities  relevant  to 
the  course 

•  Several  questions  on  the  usefulness  of  different  features  used  in  the  course 
in  terms  of  helping  the  student  to  understand  the  course  concepts, 
objectives,  and  principles;  four  questions  asking  the  students  how  much 
they  liked  some  of  the  activities  and  materials  in  the  course 

•  Four  questions  about  interactivity  between  the  students  and  the  instructor 

•  Some  other  specific  questions  included  open-ended  questions 

•  Long-Dziuban’s  Reactive  Behavior  Protocol 


There  were  five  open-ended  questions  in  this  questionnaire. 

The  first  question  asked  the  students  the  following:  ‘As  a  result  of  your  overall 

$ 

experience  in  this  course,  do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems  in  military 
operational  situations?’  Figure  40  shows  how  the  students  responded. 
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"Articulate  How  Physical  Influences  on  Space 
Systems  Impact  Our  Ability  to  Use  Them" 


Figure  40.  Distribution  of  the  Responses  to  the  First  Question. 


N  =  23 
Minimum  =  3 


Mean  =  5.65 
Maximum  =  7 


Median  =  6 
Mode  =  6 


Most  of  the  students  (nineteen  out  of  twenty-three  or  82.61%)  clearly  agreed 
they’ve  become  able  to  articulate  space  system  issues.  Responses  to  this  question 
significantly  correlate  with  affective  reactions,  the  module  readings  usefulness  question, 
self-confidence  in  discussing  and  in  accessing  resources,  and  the  quality  of  interaction 
between  the  students  and  the  instructor  (Table  40).  Seven  correlations  have  p  under  0.01, 
i.e.,  1%. 

The  second  question  was:  ‘As  a  result  of  your  experiences  in  this  course,  are  you 
able  to  articulate  how  political,  organizational  and  economic  influences  impact  how  we 
use  space  systems  in  military  operational  settings?’  Responses  to  this  question  (Figure 
41)  were  similar  to  the  first  one,  but  all  were  positive  and  only  one  was  neutral.  Seven 
correlations  were  statistically  significant  (Table  41). 
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Statement  /  Variable 

P 

p  value 

17  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.629 

0.001 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.596 

0.003 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.592 

0.003 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.585 

0.003 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.550 

0.007 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.538 

0.008 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.540 

26  -  ‘Compare  the  quality  of  interaction  between  yourself  and  the 
instructor  in  this  on-line  class  against  what  you  would  have 
expected  in  a  similar  ‘face-to-face’  class.’ 

0.429 

Table  40.  Significant  Correlations  and  Levels  of  Significance  between  the  First 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


"Articulate  Political,  Organizational,  and  Economic 
Influences  on  Use  of  Space  Systems" 


Figure  41 .  Distribution  of  the  Responses  to  the  Second  Question. 

N  =  23  Mean  =  5.83  Median  =  6 

Minimum  =  4  Maximum  =  7  Mode  =  6 
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Statement  /  Variable 

P 

p  value 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.782 

<0.001 

20  -  ‘I  am  certain  that  I  am  able  to  synthesize  information  from  a 
variety  of  sources  to  enable  me  to  use  space  systems  in  operational 

situations.’ 

0.634 

0.001 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.633 

0.001 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.585 

0.003 

1  -  ‘ .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.538 

0.008 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.494 

0.016 

1 1  -  ‘How  useful  were  the  linked  websites  in  helping  you  to 
understand  the  course  concepts,  objectives  and  principles?’ 

0.476 

0.022 

Table  4 1 .  Significant  Correlations  and  Levels  of  Significance  between  the  Second 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  third  question  was:  ‘As  a  result  of  your  experiences  in  this  course,  are  you 
able  to  articulate  how  political,  organizational  and  economic  influences  impact  how  we 
use  space  systems  in  military  operational  settings?’  Figure  42  shows  how  the  responses 
were  distributed.  Ten  students  (43.48%)  responded  with  ‘strongly  agree’,  eight  (34.78%) 
with  ‘agree’,  and  five  (21.74%)  with  ‘slightly  agree’.  None  were  neutral  or  disagreed.  Six 
correlations  were  statistically  significant  (Table  42).  The  listed  statements/questions  dealt 
with  the  issues  of  different  student  abilities  and  with  students’  view  of  the  usefulness  of 
module  readings. 
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"Articulate  Impact  of  Commercial  Space  Systems  on 
Military  Operations" 


Figure  42.  Distribution  of  the  Responses  to  the  Third  Question. 

N  =  23  Mean  =  6.21  Median  =  6 

Minimum  =  4  Max  =  7  Mode  =  7 


The  fourth  question  asked  students:  ‘Was  the  final  exam  useful  in  helping  you  to 
synthesize  the  concepts  learned  in  the  course?’  Figure  43  shows  the  responses.  Four 
students  (17.39%)  thought  the  exam  was  not  useful.  Only  three  (13.04%)  believed  the 
exam  was  extremely  useful.  Table  43  shows  the  variables  that  had  significant  correlations 
to  this  question  responses.  The  variables  listed  in  the  table  include  three  ‘affective 
reaction’  (liking)  type  questions  and  one  question  on  course  interactivity. 
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Statement  /  Variable 

P 

p  value 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.782 

<0.001 

20  -  ‘I  am  certain  that  I  am  able  to  synthesize  information  from  a 
variety  of  sources  to  enable  me  to  use  space  systems  in  operational 

situations.’ 

0.605 

0.002 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.602 

0.002 

1  -  ‘ . .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.592 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.588 

0.003 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.557 

0.006 

Table  42.  Significant  Correlations  and  Levels  of  Significance  between  the  Third 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


"Was  the  Final  Exam  Useful  in  Helping  You  to 
Synthesize  the  Concepts...?" 


Figure  43.  Distribution  of  the  Responses  to  the  Fourth  Question. 

N  =  23  Mean  =  5.13  Median  =  5 

Minimum  =  2  Max  =  7  Mode  =  6 
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Statement  /  Variable 

P 

p  value 

10  —  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.537 

0.008 

25  -  ‘Compare  the  quality  of  interaction  between  yourself  and  the 
instructor  in  this  on-line  class  against  what  you  would  have 
expected  in  a  similar  ‘face-to-face’  class.’ 

0.523 

0.011 

8  -  ‘How  useful  were  guidelines  provided  at  the  course  homepage 
for  posting  substantive  responses  to  the  Forums?’ 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.503 

0.014 

24  -  ‘Compare  the  amount  of  interaction  you  had  with  the 
instructor  for  this  on-line  course  as  compared  with  a  similar 
resident  ‘face-to-face’  course.’ 

0.456 

0.029 

17  —  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.453 

16  -  ‘How  did  you  like  the  Module  readings?’ 

15  -  ‘How  did  you  like  the  textbook  reading  assignments?’ 

0.428 

12  —  ‘How  useful  were  the  Forum  exercises  in  helping  you  to 
understand  the  subject  matter?’ 

0.422 

0.045 

Table  43.  Significant  Correlations  and  Levels  of  Significance  between  the  Fourth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  fifth  question  addressed  the  students’  confidence:  ‘I  feel  confident  in  my 
ability  to  access  appropriate  resources,-  such  as  subject  matter  experts,  documentation  of 
websites,  or  late  breaking  news  for  evaluating  uses  of  space  systems  in  operational 
situations.’  Most  of  the  students  (nine  or  39.13%)  chose  the  response  ‘agree’,  and  none 
disagreed  (Figure  44).  The  highest  correlation  this  variable  had  was  with  another  question 
about  confidence  (Table  44).  In  total,  eight  correlations  were  statistically  significant. 
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"I  Felt  Confident  in  My  Ability  to  Access  Appropriate 

Resources" 


Figure  44.  Distribution  of  the  Responses  to  the  Fifth  Question. 

N  =  23  Mean  =  5.83  Median  =  6 

Minimum  =  4  Max  =  7  Mode  =  6 

The  sixth  question  asked:  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space  systems  in  operational 
settings.’  Figure  45  shows  how  the  responses  were  distributed  and  a  slight  agreement 
(nine  responses  or  39.13%)  with  the  statement  dominated.  Table  45  lists  seven  variables 
that  significantly  correlate  with  the  responses  to  this  statement. 
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Statement  /  Variable 

P 

p  value 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.736 

<  0.001 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.557 

0.006 

1  -  ‘ .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.540 

0.008 

20  -  ‘I  am  certain  that  I  am  able  to  synthesize  information  from  a 
variety  of  sources  to  enable  me  to  use  space  systems  in  operational 

situations.’ 

0.509 

0.013 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.494 

0.016 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.482 

0.020 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept...’ 

0.444 

0.034 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.436 

0.038 

Table  44  Significant  Correlations  and  Levels  of  Significance  between  the  Fifth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


"I  Felt  Confident  in  My  Ability  to  Get  Involved  in 
Open  Discussions  with  My  Peers..." 


10 
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7 

Number  of  | 
students  4 
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Figure  45.  Distribution  of  the  Responses  to  the  Sixth  Question. 

N  =  23  Mean  =5.65  Median  =  6 


Minimum  =  4 


Max  =  7 
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Mode  =  5 


Statement  /  Variable 

P 

p  value 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.736 

<0.001 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.633 

0.001 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.602 

0.002 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.564 

0.005 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.555 

0.006 

1  -  ‘ .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.550 

0.007 

20  -  ‘I  am  certain  that  I  am  able  to  synthesize  information  from  a 
variety  of  sources  to  enable  me  to  use  space  systems  in  operational 

situations.’ 

0.478 

0.021 

Table  45.  Significant  Correlations  and  Levels  of  Significance  between  the  Sixth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  seventh  question  and  the  distribution  of  the  responses  are  in  Figure  46.  Asked 
about  usefulness  of  the  guidelines  on  the  course  homepage,  few  students  (two  or  8.70%) 
chose  the  response  ‘useless’,  but  a  majority  of  the  students  (twenty  students  or  86.96%) 
saw  the  guidelines  as  useful.  Only  one  correlation  was  significant  (Table  46),  and  that 
was  with  responses  to  the  question  asking  about  guidelines. 
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"How  Useful  Were  the  Guidelines  Provided  in  the 
Course  Homepage  for  Your  Ability  to  Succeed?" 


Figure  46.  Distribution  of  the  Responses  to  the  Seventh  Question. 


N  =  23  Mean  =  5.61  Median  =  6 

Minimum  =  2  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

8  -  ‘How  useful  were  guidelines  provided  at  the  course  homepage 
for  posting  substantive  responses  to  the  Forums?’ 

0.711 

<0.001 

Table  46.  Significant  Correlation  and  Level  of  Significance  between  the  Seventh 

Question  and  Another  Variable. 

Student  responses  to  the  second  question  about  the  guidelines  (‘How  useful  were 
guidelines  provided  at  the  course  homepage  for  posting  substantive  responses  to  the 
Forums?’)  distributed  differently  than  the  first  question  (Figure  47),  with  more  neutral 
responses  (five  or  21.74%)  and  a  higher  frequency  of  ‘useful’  responses  (thirteen  or 
56.53%).  Four  correlations  appeared  to  be  significant  (Table  47).  Interestingly,  the 
responses  to  questions  about  module  readings,  quizzes,  and  final  exams  have  significant 
correlations  with  responses  to  this  question  but  to  none  of  the  other  variables.  This  is 
difficult  to  explain. 
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"How  Useful  Were  the  Guidelines  for  Posting 
Substantive  Responses  to  the  Forums?" 


Figure  47.  Distribution  of  the  Responses  to  the  Eighth  Question. 

N  =  23  Mean  =  5.30  Median  =  6 

Minimum  =  2  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

p  value 

7  -  ‘How  useful  were  the  guidelines  provided  in  the  course 
homepage  to  your  ability  to  succeed?’ 

0.711 

<0.001 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.516 

0.012 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.506 

0.014 

13  -  ‘How  useful  were  the  Quizzes  (2, 3/4/5, 6/7)  for  providing 
feedback  in  your  understanding  of  the  material  for  those  Modules?’ 

0.500 

0.015 

Table  47.  Significant  Correlations  and  Levels  of  Significance  between  the  Eighth 

Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

i 

The  ninth  question  asked  students  about  the  usefulness  of  the  textbook:  ‘How 
useful  was  the  textbook  in  helping  you  understand  the  course  concepts,  objectives  and 
principles?’  Figure  48  shows  the  distribution  of  the  student  responses.  Most  of  the 
students  (twenty-one  students  or  91.30%)  saw  the  textbook  as  useful,  but  there  was  one 
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response  ‘useless’,  and  one  neutral  response.  Only  one  correlation  was  significant  (Table 
48). 


"How  Useful  Was  the  Textbook  in  Helping  You 
to  Understand  the  Course  Concepts...?" 


Figure  48.  Distribution  of  the  Responses  to  the  Ninth  Question. 

N  =  23  Mean  =  6.13  Median  =  6 

Minimum  =  2  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

15  -  ‘How  did  you  like  the  textbook  reading  assignments?’ 

0.636 

0.001 

Table  48.  Significant  Correlation  and  Level  of  Significance  between  the  Ninth 

Question  and  Another  Variable. 

Most  students  (fourteen  students  or  60.87%)  saw  the  on-line  module  readings  as 
‘extremely  useful’  (Figure  49).  Ten  correlations  were  significant.  The  correlations  and 
significance  levels  are  in  Table  49. 
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Figure  49.  Distribution  of  the  Responses  to  the  Tenth  Question. 


N  =  23  Mean  =  6.57  Median  =  7 

Minimum  =  7  Max  =  7  Mode  =  7 

The  fifth  question  asked  students:  ‘How  useful  were  the  linked  websites  in 
helping  you  to  understand  the  course  concepts,  objectives,  and  principles?’  Most  students 
chose  to  respond  with  ‘useful’  (eleven  students  or  47.83%),  and  slightly  less  (ten  or 
43.48%)  with  ‘extremely  useful’  (Figure  50). 

Only  three  correlations  were  significant  (Table  50).  The  highest  correlation  was 
with  another  question  related  to  liking  the  links  feature  the  course  provided.  This 
correlation  of  approximately  0.56  means  that  a  utility  assessment  (thinking  of  usefulness 
of  a  feature)  and  an  affective  reaction  (liking  it)  have  a  significant  association  but  are  not 
identical.  Otherwise  the  correlation  would  be  much  higher,  close  to  1.  Two  other 
correlations  are  difficult  to  interpret.  They  are  correlations  with  the  student  responses  to 

statements/questions  related  to  several  of  the  course  learning  objectives. 
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Statement  /  Variable 

. _  P  _ 

p  value 

17  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.644 

1  -  ‘ . .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.596 

0.003 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.588 

0.003 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.585 

0.003 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.564 

0.005 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.541 

0.008 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.537 

0.008 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.536 

0.008 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.512 

0.012 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.444 

0.034 

Table  49.  Significant  Correlations  and  Levels  of  Significance  between  the  Tenth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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"How  Useful  Were  the  Linked  Websites  in 
Helping  You  to  Understand  the  Course 
Concepts...?” 
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Figure  50.  Distribution  of  the  Responses  to  the  Eleventh  Question. 

N  =  23  Mean  =  6.35  Median  =  6 

Minimum  =  5  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

p  value 

1 7  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.557 

WE  AM 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.476 

0.022 

20  - ‘I  am  certain  that  I  am  able  to  synthesize  information  from  a 
variety  of  sources  to  enable  me  to  use  space  systems  in  operational 

situations.’ 

0.475 

0.022 

Table  50.  Significant  Correlations  and  Levels  of  Significance  between  the  Eleventh 

Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  twelfth  question  in  the  Final  Survey  questionnaire  was  about  the  usefulness 
of  the  forum  exercises.  Responses  to  this  question  vary  from  ‘useless’  (three  students  or 
13.04%)  to  ‘extremely  useful’  (one  student  or  4.35%),  with  responses  ‘useful’ 
dominating  (eight  responses  or  34.78%).  Variability  of  the  responses  is  larger  than  in  the 

previous  questions  from  this  set  (Figure  51).  Similar  to  die  eleventh  question,  this 
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question  had  two  correlations.  The  highest  correlation  was  with  the  response  to  the 
question  about  liking  the  forums  (Table  51). 


"How  Useful  Were  the  Forum  Exercises  in  Helping 
You  to  Understand  the  Subject  Matter?" 


Figure  5 1 .  Distribution  of  the  Responses  to  the  Twelfth  Question. 

N  =  23  Mean  =  4.65  Median  =  5 


Minimum  =  2  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

14  -  ‘How  did  you  like  the  Forum  assignments?’ 

0.620 

0.002 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.422 

0.045 

Table  5 1 .  Significant  Correlations  and  Levels  of  Significance  between  the  Twelfth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  instructor  recommended  the  next  interesting  question  to  determine  if  the 

quizzes  in  the  course  served  the  purpose  she  had  intended:  as  feedback  to  the  students. 

Figure  52  shows  the  distribution  of  the  responses.  Most  of  the  students  (seventeen  or 

73.91%)  saw  the  quizzes  as  more  or  less  useful  as  feedback,  but  four  (17.39%)  had  the 

opposite  opinion.  Two  (8.70%)  chose  neutral  responses.  Five  correlations  were 

significant,  but  none  above  0.50.  It  is  difficult  to  understand  why  this  variable  correlates 
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with  variables  dealing  with  module  readings,  interactivity  between  the  students,  and  the 
flexibility  that  helped  in  succeeding  in  other  classes  (Table  52). 


"How  Useful  Were  the  Quizzes  for  Providing 
Feedback  in  Your  Understanding  of  the  Material...?" 


Figure  52.  Distribution  of  the  Responses  to  the  Thirteenth  Question. 

N  =  23  Mean  =  4.78  Median  =  6 

Minimum  =  2  Max  =  7  Mode  =  6 


Statement  /  Variable 

P 

p  value 

8  -  ‘How  useful  were  guidelines  provided  at  the  course  homepage 
for  posting  substantive  responses  to  the  Forums?’ 

0.500 

0.015 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.492 

0.017 

27  —  ‘Compare  the  quality  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.492 

0.017 

1 6  -  ‘How  did  you  like  the  Module  readings?’ 

0.463 

0.026 

17  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.442 

0.035 

Table  52.  Significant  Correlations  and  Levels  of  Significance  between  the  Thirteenth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  next  question  asked  the  students  how  they  liked  forum  assignments  (Figure 

53).  Most  of  them  ‘slightly  liked’  (eight  students  or  34.78%)  and  ‘liked’  (six  students 
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26.09%)  them,  but  four  students  (17.39%)  gave  negative  responses.  Five  (21.74%)  gave 
neutral  responses.  Only  two  correlations  were  significant:  one  with  responses  to  the 
question  about  the  usefulness  of  forum  exercises  which  is  a  very  logical  relationship,  and 
another,  about  the  amount  of  interactivity  between  the  students  and  the  instructor.  The 
students,  who  saw  the  forums  as  useful,  liked  them.  Those  who  thought  that  the  course 
provided  a  satisfactoiy  amount  of  interaction  compared  to  face-to-face  classes  with  the 
instructor  tended  to  like  forum  assignments  more  than  the  students  unsatisfied  with  the 
amount  of  interaction  (Table  53). 


"How  did  you  Like  the  Forum  Assignments?" 


Figure  53.  Distribution  of  the  Responses  to  the  Fourteenth  Question. 

N  =  23  Mean  =  4.56  Median  =  5 

Minimum  =  1  Max  =  6  Mode  =  5 
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Statement  /  Variable 

P 

p  value 

12  -  ‘How  useful  were  the  Forum  exercises  in  helping  you  to 
understand  the  subject  matter?’ 

0.620 

24  -  ‘Compare  the  amount  of  interaction  you  had  with  the 
instructor  for  this  on-line  course  as  compared  with  a  similar 
resident  ‘face-to-face’  course.’ 

0.487 

0.018 

Table  53.  Significant  Correlations  and  Levels  of  Significance  between  the 
Fourteenth  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


Figure  54  shows  the  distribution  of  student  responses  about  liking  the  textbook 
assignments.  Most  of  them  (eleven  students  or  47.83%)  only  ‘slightly  liked’  them,  a 
substantial  number  (seven  students  or  30.43%)  ‘liked’  them,  but  two  students  (8.70%) 
disliked  them,  while  three  (13.04%)  were  neutral.  Table  54  shows  three  significant 
correlations  this  variable  had  with  responses  to  the  question  about  the  usefulness  of  the 
textbook,  usefulness  of  the  final  exam,  and  about  students’  feeling  about  how  well  this 
this  course  flexibility  helped  them  for  success  in  other  courses.  The  last  two  correlations 
are  not  easy  to  explain,  while  the  first  relationship  is  very  understandable. 


"How  did  you  Like  the  Textbook  Reading 
Assignments?" 


Figure  54.  Distribution  of  the  Responses  to  the  Fifteenth  Question. 
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N  =  23 
Minimum  =  1 


Mean  =  4.91 
Max  =  6 


Median  =  5 
Mode  =  5 


Statement  /  Variable 

P 

p  value 

9  -  ‘How  useful  was  the  textbook  in  helping  you  understand  the 
course  concepts,  objectives  and  principles?’ 

0.636 

0.001 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.428 

0.042 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.425 

0.043 

Table  54.  Significant  Correlations  and  Levels  of  Significance  between  the  Fifteenth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


'How  did  You  Like  the  Module  Readings?' 
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Figure  55.  Distribution  of  the  Responses  to  the  Sixteenth  Question. 


N  =  23 

Minimum  =  2 


Mean  =  5.34 
Max  =  6 


Median  =  6 
Mode  =  6 


The  next  question  asked  the  students  how  much  they  liked  the  module  readings. 
Over  60%  of  the  students  chose  response  ‘liked’,  and  almost  all  others  ‘slightly  liked’  the 
readings(Figure  55).  Ten  correlations  were  significant,  starting  with  questions  on  liking 
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website  links,  beneficial  flexibility  of  the  course,  and  usefulness  of  the  module  readings 
(Table  55).  Other  questions  included  issues  like  amount  of  interactivity  with  other 
students,  quizzes,  and  final  exams. 


Statement  /  Variable 

P 

p  value 

1 7  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.632 

0.001 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.590 

0.003 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.541 

Q.008 

26  -  ‘Compare  the  amount  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.539 

0.008 

8  -  ‘How  useful  were  guidelines  provided  at  the  course  homepage 
for  posting  substantive  responses  to  the  Forums?’ 

0.516 

0.012 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.480 

0.020 

13  -  ‘How  useful  were  the  Quizzes  (2, 3/4/5, 6/7)  for  providing 
feedback  in  your  understanding  of  the  material  for  those  Modules?’ 

0.463 

0.026 

25  -  ‘Compare  the  quality  of  interaction  between  yourself  and  the 
instructor  in  this  on-line  class  against  what  you  would  have 
expected  in  a  similar  ‘face-to-face’  class.’ 

0.439 

0.036 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.436 

0.038 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.434 

0.039 

Table  55.  Significant  Correlations  and  Levels  of  Significance  between  the  Sixteenth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  question  about  liking  website  links  provided  in  the  course  produced  very 
favorably  distributed  responses  (Figure  56).  The  response  ‘strongly  liked’  dominated 
(twelve  students  or  52.17%)  and  the  response  ‘liked’  followed  (eight  students  or 
34.78%).  There  were  no  negative  reactions,  and  only  one  neutral  response.  Table  56 

shows  nine  significant  correlations.  Responses  to  questions  about  the  amount  of 
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interaction  between  the  students,  about  liking  the  module  readings,  usefulness  of  module 
readings,  class  flexibility,  etc,  are  among  significantly  correlationg  variables. 


Figure  56.  Distribution  of  the  Responses  to  the  Seventeenth  Question. 


N  =  23  Mean  =  6.35  Median  =  7 

Minimum  =  4  Max  =  7  Mode  =  7 
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Statement  /  Variable 

P 

p  value 

26  -  ‘Compare  the  amount  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.633 

0.001 

1 6  -  ‘How  did  you  like  the  Module  readings?’ 

0.632 

0.001 

1  -  ‘ .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.629 

0.001 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.614 

0.001 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in  other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.563 

0.005 

1 1  -  ‘How  useful  were  the  linked  websites  in  helping  you  to 
understand  the  course  concepts,  objectives  and  principles?’ 

0.557 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.528 

0.010 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.453 

0.030 

13  -  ‘How  useful  were  the  Quizzes  (2, 3/4/5, 6/7)  for  providing 
feedback  in  your  understanding  of  the  material  for  those  Modules?’ 

0.442 

0.035 

Table  56.  Significant  Correlations  and  Levels  of  Significance  between  the 
Seventeenth  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  eighteenth  question  was  one  of  five  open-ended  questions  in  this  instrument. 
It  asked:  ‘What  other  instructional  materials,  if  any,  should  be  included  in  the  course?’ 
Five  students  did  not  respond.  The  responses  of  the  others  follow: 

‘None*  -  four  opinions 


•  Can't  think  of  any 

•  N/a  for  an  online  course:  two  responses 

•  Nothing  that  I  can  see  at  this  time 

•  The  materials  that  were  provided  were  adequate 

Suggestions 

•  A  classroom 

•  Better  textbook.  Classroom  discussion. 
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•  I  think  the  instructor  somehow  needs  to  play  a  bigger  role.  The  students 
are  not  the  subject  matter  experts  so  it  got  a  little  old  listening  to  their 
comments.  I  would  prefer  to  hear  from  the  subject  matter  expert. 

•  More  articles,  etc.  That's  where  I  got  a  better  feel  for  the  topics. 

•  Perhaps  more  ‘expert’  participation  in  the  forums.  The  information 
provided  by  LCDR  Z.  was  very  useful. 

•  References  to  periodicals  that  the  library  is  on  distribution  for 

•  Search  capabilities  for  forums 

•  The  Army  Handbook  should  be  available  either  in  hardcopy  or  on  CD- 
ROM 


Other  notes 

•  I  thought  the  textbook  was  very  good,  but  expensive 


The  next  open-ended  statement  asked:  ‘Describe  your  reaction  to  the  instructional 
strategies  used  to  teach  the  subject  matter  (textbook  readings,  Forums,  websites)’.  Four 
students  did  not  respond.  The  responses  were  as  follows: 


Positive  reactions 


•  Excellent 

•  Good 

•  I  approve  and  have  no  comments  on  these  strategies 

•  I  liked  it  very  much.  See  questions  1 8  and  21 . 

•  Learning  conceptual  ideas  were  well  suited  for  this  particular  online 
course 

•  No  real  difficulties 

•  The  modular  readings  were  great.  Short  and  to  the  point.  You  need  more 
tests  to  force  one  to  at  least  get  some  understanding  out  of  it.  Fewer 
forums. 

•  The  module  readings  and  websites  links  greatly  simplified  the  complex 

concepts  provided  in  the  text 

•  They  were  all  sufficient 

•  They  were  fine.  I  can't  think  of  a  better  way  to  do  it. . . 
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•  Too  much  flexibility 

Neutral  remarks 

•  Requires  self-motivation 

•  I  didn't  read  much  of  the  forum  discussions  due  to  a  lack  of  time  and 
priorities 

•  Thought  they  were  good,  but  honestly  don't  know  how  much  good  the 
quizzes  did  -  i.e.  someone  could  skip  the  readings  and  just  look  up  the 
answers.  It's  really  up  to  the  students  to  do  the  reading  if  he  wants  to  learn 
-  and  this  is  grad  school,  so  maybe  that's  the  way  it  should  be? 

•  Same  as  question  1 8.  (The  response  to  question  1 8  was  ‘None’.) 

Negative  reactions  and  critiques 

•  Flexibility  was  not  built  into  this  course.  Modules  were  not  completely 
written  prior  to  the  course  beginning  and  they  were  not  available  until  the 
instructor  opened  them.  That  reduces  the  opportunity  to  work  ahead. 

•  In  a  remote  environment  it  would  be  great.  However,  at  a  university  it 
does  not  work  for  me. 

•  Textbook  -  too  technical.  Websites  -  better  than  textbook.  Best  part  of 
course  forums  -  dull. 

•  Textbook  was  terrible.  The  forums  might  have  been  useful,  but  after 
awhile  it  seemed  as  if  we  kept  talking  about  the  same  things. 


The  twentieth  question  asked:  ‘  I  am  certain  that  I  am  able  to  synthesize 
information  from  a  variety  of  sources  to  enable  me  to  use  space  systems  in  operational 
situations.’  As  Figure  57  shows,  most  of  the  student  ‘slightly  agreed’  (ten  students  or 
45.45%)  or  ‘agreed’  (eight  students  or  36.36%)  with  this  statement.  Only  two  (9.09%) 
‘slightly  disagreed’.  Table  57  shows  five  correlations,  which  were  significant  with  two 
other  statements  addressing  student  abilities  to  articulate  certain  subject  area  issues, 
ability  to  access  appropriate  resources,  ability  to  participate  in  open  discussions,  and  the 
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usefulness  of  linked  websites  to  help  a  student  understand  the  course  concepts, 
objectives,  and  principles. 


"I  Am  Certain  that  I  Am  Able  to  Synthesize 
Information  from  a  Variety  of  Sources..." 


rtf®' 


& 


rtf®B 


No  response:  one 
student 


Figure  57.  Distribution  of  the  Responses  to  the  nineteenth  Question. 

N  =  22  Mean  =  5.36  Median  =  5 

Minimum  =  3  Max  =  7  Mode  =  5 


Statement  /  Variable 

P 

p  value 

2  -  ‘As  a  result  of  your  experiences  in  this  course,  are  you  able  to 
articulate  how  political,  organizational  and  economic  influences 
impact  how  we  use  space  systems  in  military  operational  settings? 

0.634 

0.002 

3  -  ‘Do  you  feel  that  you  are  able  to  articulate  how  commercial 
space  systems  will  impact  military  operations?’ 

0.605 

0.002 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.509 

0.013 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.478 

0.021 

1 1  -  ‘How  useful  were  the  linked  websites  in  helping  you  to 
understand  the  course  concepts,  objectives  and  principles?’ 

0.475 

0.022 

Table  57.  Significant  Correlations  and  Levels  of  Significance  between  the  Twentieth 
Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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The  twenty-first  question  was  also  open-ended:  “As  a  result  of  this  on-line  course 
experience,  do  you  feel  that  you  have  changed  your  fundamental  approach  to  learning? 
Please  respond  ‘yes’,  ‘somewhat’  or  ‘no’,  and  explain  your  answer.”  Clearly,  most  of  the 
students  disagreed  they  changed  their  fundamental  approach  to  learning.  Below  are  the 
systematized  student  responses: 


Yes 

•  >  Yes,  I  have  learned  that  there  is  a  great  deal  of  information  available 

through  many  sources  and  media.  The  Internet,  texts,  periodicals,  and 
instructor's  modules  all  contributed  to  a  great  learning  experience. 

•  Yes,  I  plan  on  researching  distant  learning  for  a  possible  thesis  subject 

•  Yes,  only  because  I  am  aware  of  many  of  the  opportunities  provided  by 
the  NETG  program  and  would  feel  more  comfortable  going  through  some 
of  those  online  courses. 

Somewhat 

•  Somewhat  no.  Still  miss  the  students’  interaction. 

•  Somewhat 

•  Somewhat.  I  still  think  that  you  can’t  fully  replace  the  added  value  with  an 
instructor  in  a  classroom  environment.  However,  I  feel  that  adequate  and 
interesting  learning  can  take  place  on-line. 

•  Somewhat.  Think  on-line  is  a  good  venue  for  certain  types  of 
courses/curricula.  The  Space  Tech  course  for  ITM  is  such  a  combination. 

No 

•  No,  because  I  feel  that  I  was  already  receptive  to  the  distance  learning 
concept  of  education  having  just  completed  another  18  months  long 
masters  program  with  a  different  university. 

•  No,  I  need  the  classroom  setting 

•  No,  I  still  like  to  take  notes,  and  have  verbal  discussion  in  a  short  period  of 

time,  instead  of  taking  an  hour(s)  explaining/writing  a  message  to  post  to 
the  forum. 

•  No,  I'm  hard  headed 


145 


•  No.  (Three  responses.) 

•  No.  I  did  what  I  had  to  do  to  fulfill  the  requirements  of  the  course.  Where  I 
was  more  interested,  I  put  more  effort  into  exploring  the  topic  areas.  This 
is  how  I  approach  learning  in  any  situation.  I  don't  feel  like  that  has 
changed  because  of  this  course. 

•  No.  I  have  always  searched  for  additional  resources  to  expand  on 
presented  material.  I  like  to  form  unbiased  and  objective  opinions  on 
subjective  matter.  Also,  I  like  to  see  different  explanations  on  scientific 
matters  to  ensure  understanding.  This  course  reinforced  that  approach. 

•  No.  I  shall  prefer  classroom  environment  with  one-to-one  approach. 

•  No.  I  still  feel  I  would  get  more  out  of  in  class  instruction.  Perhaps  that  is 
just  a  personal  preference.  The  basic  concept  here  is  that  you  should  teach 
yourself  via  the  online  class,  which  is  probably  ok  for  something  like  a 
basic  space  class.  But  I  would  rather  have  a  professor  teach  me  face-to- 
face,  allowing  for  in-class  discussion,  and  focus  of  concepts  and  efforts. 

•  No.  I  think  this  method  lends  itself  to  training  type  courses,  not  education. 

•  No.  I  thought  going  in  that  this  would  be  difficult  because  I  tend  to  get  the 
most  out  of  class  discussions  vice  reading,  but  wanted  to  give  a  try.  I  am 
now  more  confident  than  ever  that  DL  is  not  for  me. 

•  No . . .  My  ways  of  learning  are  pretty  well  set. 

•  No. . .  Read  and  absorb  is  still  primary... 

•  Not  yet  - 1  did  not  put  my  best  forward  this  quarter,  in  this  class. 


In  question  23  of  A-2  questionnaire,  with  question  23  (‘So  far,  what  do  you  like 
about  this  on-line  course?’),  most  students  responded  they  liked  the  time  management 
flexibility  of  the  course.  The  next  two  questions  in  the  Final  survey  readdressed  that 
issue:  one  in  the  context  of  student  success  within  the  course  and  the  another  in  the  sense 
of  student  success  in  other  classes/courses.  Figure  58  shows  the  distribution  of  the 
responses  to  the  first  question  (‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’).  The  majority  of  the  students  ‘strongly 
agreed’  (eight  students  or  34.78%)  or  ‘agreed’  (seven  students  or  30.43%),  but  six  of 
them  (26.09%)  disagreed.  Table  58  shows  the  significant  correlations.  There  were  seven 
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of  them:  liking  the  module  readings,  quality  of  interaction  with  the  instructor,  liking  the 
website  links,  and  with  three  questions  dealing  with  self-perceived  abilities  or 
confidence. 


"Did  You  See  the  Flexibility  of  this  On¬ 
line  Course  as  Beneficial  as  Your 
Ability  to  Succeed...?" 


Figure  58.  Distribution  of  the  Responses  to  the  Twenty-Second  Question. 


N  =  23  Mean  =  5.30  Median  =  6 

Minimum  =  2  Max  =  7  Mode  =  7 


Statement  /  Variable 

P 

p  value 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.590 

0.003 

1  -  ‘ .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.585 

0.003 

6  -  ‘I  feel  confident  in  my  ability  to  get  involved  in  open 
discussions  with  my  peers  to  weigh  issues  related  to  using  space 
systems  in  operational  settings.’ 

0.555 

0.006 

25  -  ‘Compare  the  quality  of  interaction  between  yourself  and  the 
instructor  in  this  on-line  class  against  what  you  would  have 
expected  in  a  similar  ‘face-to-face’  class.’ 

0.533 

0.009 

17  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.528 

0.010 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.512 

0.012 

5  -  ‘I  feel  confident  in  my  ability  to  access  appropriate  resources, 
such  as  subject  matter  experts,  documentation  of  websites,  or  late 
breaking  news  for  evaluating  uses  of  space  systems  in  operational 

situations.’ 

0.482 

0.020 

Table  58.  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Second  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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The  second  question  about  flexibility  had  a  different  distribution  of  the  responses 
than  the  first  one  (Figure  59):  asked  if  the  flexibility  hepled  them  succeed  in  other 
clasees,  all  students  agreed  with  the  statement.  Ten  of  them  chose  ‘agree’  and  ten 
‘strongly  agree’  (43.48%  each).  Table  59  shows  the  correlations  with  other  variables. 
Seven  were  significant.  Only  two  variables  appeared  significantly  correlated  with 
responses  to  both  questions  about  the  helpfulness  of  the  flexibility  of  the  course:  liking 
website  links  and  liking  module  readings. 


"Do  You  Feel  that  the  Flexibility  of  this  Class  Helped 

You  Suceed  in  Other  Classes...?" 

1  W 

ft 

Number  of  ° 
students  4. 

4' 

|^P  l  . -  mm, 

<  1  1 - : - 1 - r . . 1 - 1 

-x6e  rfee  ,ee  -<e6  rt<e®  A<ee 
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Figure  59.  Distribution  of  the  Responses  to  the  Twenty-Third  Question. 


N  =  23 

Minimum  =  5 


Mean  =  6.21 
Max  =  7 


Median  =  6 
Mode  =  7 
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Statement  /  Variable 

P 

p  value 

1 7  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.563 

0.005 

10  -  ‘How  useful  were  the  on-line  Module  readings  in  helping  you 
to  understand  the  course  concept. . .  ’ 

0.536 

0.008 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.503 

0.014 

26  -  ‘Compare  the  amount  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.500 

0.015 

13  -  ‘How  useful  were  the  Quizzes  (2, 3/4/5, 6/7)  for  providing 
feedback  in  your  understanding  of  the  material  for  those  Modules?’ 

0.492 

0.017 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.480 

0.020 

1 5  -  ‘How  did  you  like  the  textbook  reading  assignments?’ 

0.425 

0.043 

Table  59.  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Third  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 


The  twenty-fourth  item  was  the  first  of  four  dealing  with  issues  of  interactivity  in 
the  course.  The  first  question  asked  students  to:  “Compare  the  amount  of  interaction  you 
had  with  the  instructor  for  this  on-line  course  as  compared  with  a  similar  resident  ‘face- 
to-face’  course.  Note  that  this  question  is  about  ‘amount’  while  another  question  will 
cover  ‘quality  of  interaction’.  Try  to  consider  only  the  amount  of  interaction  that  occurred 
on-line  (including  e-mails)  as  opposed  to  what  occurred  if/when  you  stopped  by  the 
instructor's  office.” 

Figure  60  shows  the  distribution  of  the  responses:  most  of  the  students  felt  there 
were  less  interaction  between  the  students  and  the  instructor:  ten  out  of  twenty-two 
(43.48%)  responded  ‘significantly  less’;  five  (21.74%)  responded  ‘less’;  and  two  (8.70%) 
responded  ‘slightly  less’.  Four  responses  (17.39%)  were  in  the  middle  of  the  scale  (‘about 
the  same’)  and  two  on  the  opposite  end  (one  [4.35%]  responded  ‘slightly  more’,  and  one 
[4.35%]  responded  ‘more’).  This  variable  had  significant  correlations  with  only  three 
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other  variables  (Table  60).  All  three  of  them  actually  had  some  kind  of  interaction 
between  the  students  and  the  instructor:  quality  of  interaction  between  the  students  and 
the  instructor;  liking  forum  assignments  (in  forums  the  instructor  also  interacted  with  the 
students);  and  students  view  on  usefulness  of  the  final  exam. 


Amount  of  Interaction  Between  the  Student  and  the 
Instructor  in  the  On-line  Course  Compared  to  a  Face- 

101 

8 

Number  of  6 
students  4 

O. 

to-face  Course 
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Figure  60.  Distribution  of  the  Responses  to  the  Twenty-Fourth  Statement. 


N  =  23  Mean  =  2.30  Median  =  2 

Minimum  =  1  Max  =  6  Mode  =  1 


Statement  /  Variable 

P 

p  value 

25  -  ‘Compare  the  quality  of  interaction  between  yourself  and  the 
instructor  in  this  on-line  class  against  what  you  would  have 
expected  in  a  similar  ‘face-to-face’  class.’ 

0.666 

0.001 

14  -  ‘How  did  you  like  the  Forum  assignments?’ 

0.487 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.456 

0.029 

Table  60.  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Fourth  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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The  next  statement  about  interactivity  was  the  following:  “Compare  the  quality  of 
interaction  between  yourself  and  the  instructor  in  this  on-line  class  against  what  you 
would  have  expected  in  a  similar  "face-to-face"  class.” 

Figure  61  shows  the  distribution  of  the  student  responses.  This  distribution  is 
different  from  the  distribution  of  student  responses  to  the  question  about  the  amount  of 
interaction.  In  this  distribution  there  were  six  (26.09%)  ‘significantly  less’  responses  and 
eleven  (47.83%)  ‘about  the  same’  responses,  while  in  previous  question  there  were 
eleven  (47.83%)  ‘significantly  less  ’  responses  and  four  (17.39%)  ‘about  the  same’ 
responses.  . 


Table  61  shows  significant  correlations. 


Quality  of  Interaction  Between  the  Student  and  the 
Instructor  in  the  On-line  Course  Compared  to  a  Face- 
to-face  Course 


Number  of 
students 


✓ 


Figure  61 .  Distribution  of  the  Responses  to  the  Twenty-Fifth  Statement. 

N  =  23  Mean  =  3.09  Median  =  4 


Minimum  =  1  Max  =  6  Mode  =  4 
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Statement  /  Variable 

P 

p  value 

24  -  ‘Compare  the  amount  of  interaction  you  had  with  the 
instructor  for  this  on-line  course  as  compared  with  a  similar 
resident  ‘face-to-face’  course.’ 

0.666 

0.001 

22  -  ‘Did  you  view  the  flexibility  of  this  on-line  course  as 
beneficial  to  your  ability  to  succeed  in  this  class?’ 

0.533 

0.009 

4  -  ‘Was  the  final  exam  useful  in  helping  you  to  synthesize  the 
concepts  learned  in  the  course?’ 

0.523 

0.011 

1 6  -  ‘How  did  you  like  the  Module  readings?’ 

0.439 

0.036 

Table  6 1 .  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Fifth  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  twenty-sixth  statement,  which  was  the  third  statement  on  interactivity,  was: 
“Compare  the  amount  of  interaction  you  had  with  other  students  for  this  on-line  course 
with  a  similar  resident  ‘face-to-face’  course.  Note  that  this  question  is  about  ‘amount’ 
while  another  question  will  cover  ‘quality  of  interaction’.” 

Figure  62  shows  the  distribution  of  responses.  The  responses  seemed  to  group 
around  two  areas:  firstly  around  ‘significantly  less’  and  ‘less’  (their  total  is  twelve 
students  or  52.17%),  and  secondly  around  ‘about  the  same’  and  ‘slightly  more’  (nine 
students  or  39.13%).  Although  almost  all  the  responses  were  present  (six  of  seven 
possible)  and  student  responses  were  split  into  two  groups  with  opposite  opinions, 
students  predominantly  thought  that  amount  of  the  interaction  in  the  course  was  lower 
than  in  a  face-to-face  course.  Students  significantly  differ  by  perception  of  the  interaction 
when  comparing  DL  to  face-to-face  course,  what  is  probably  significant  factor  of  their 
reactions  about  the  course.  However,  this  finding  does  not  tell  us  anything  about  the 
cause  of  such  split  in  opinions. 
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Amount  of  Interaction  Between  the  Students  in  the 

On-line  Course  Compared  to  a  Face-to-face  Course 
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Figure  62.  Distribution  of  the  Responses  to  the  Twenty-Sixth  Statement. 

N  =  23  Mean  =  2.96  Median  =  2 

Minimum  =  1  Max  =  6  Mode  =  2 


This  variable  had  six  statistically  significant  correlations  (Table  62),  with  the 
following  variables:  liking  website  links,  liking  module  readings,  opinion  about  the  help 
the  flexibility  of  this  course  provide  to  succeed  in  other  classes,  quality  of  the  interaction 
between  students,  and  others. 


Statement  /  Variable 

P 

p  value 

17  -  ‘How  did  you  like  the  website  links  that  were  provided?’ 

0.633 

0.001 

16  -  ‘How  did  you  like  the  Module  readings?’ 

0.539 

0.008 

27  -  ‘Compare  the  quality  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.511 

0.013 

23  -  ‘Do  you  feel  that  the  flexibility  of  this  class  helped  you 
succeed  in.other  classes  that  you  were  taking  in  the  same  quarter?’ 

0.500 

0.015 

1  -  ‘ . .  .Do  you  feel  that  you  are  able  to  articulate  how  physical 
influences  on  space  systems  impact  our  ability  to  use  those  systems 
in  military  operational  situations?’ 

0.429 

0.041 

Table  62.  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Sixth  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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The  next  question  asked:  ‘Compare  the  quality  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident  ’ace-to-face’  course.’ 

The  students  gave  approximately  the  same  number  of  responses  to  each  category 
from  ‘significantly  less’  (five  or  21.75%)  to  ‘about  the  same’  (also  five),  what  makes 
total  of  eighteen  students  (78.26%).  Fewer  students  (21.74%)  thought  the  quality  was 
better  in  the  on-line  class  (three  students  [13.04%])  responded  ‘slightly  greater  quality’, 
one  (4.35%)  responded  ‘greater’,  and  one  ‘significantly  greater’).  Figure  63  shows  the 
distribution.  Only  two  correlations  were  significant:  with  the  amount  of  that  same 
interaction  between  the  students,  and  with  their  opinions  on  the  usefulness  of  the  quizzes 
(Table  63). 


Quality  of  Interaction  Between  the  Students  in  the 

On-line  Course  Compared  to  a  Face-to-face  Course 
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Figure  63.  Distribution  of  the  Responses  to  the  Twenty-Seventh  Statement. 

N  =  23  Mean  =  3.17  Median  =  2 

Minimum  =  1  Max  =  6  Mode  =  2 
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Statement  /  Variable 

P 

p  value 

26  -  ‘Compare  the  amount  of  interaction  you  had  with  other 
students  for  this  on-line  course  as  compared  with  a  similar  resident 
‘face-to-face’  course.’ 

0.511 

0.013 

13  -  ‘How  useful  were  the  Quizzes  (2, 3/4/5, 6/7)  for  providing 
feedback  in  your  understanding  of  the  material  for  those  Modules?’ 

0.492 

0.017 

Table  63.  Significant  Correlations  and  Levels  of  Significance  between  the  Twenty- 
Seventh  Question  and  Other  Variables  (Ordered  by  Level  of  Significance). 

The  twenty-eighth  question  was  open-ended.  It  asked  the  following:  ‘Describe 
your  reaction  to  the  media  used  to  present  the  instruction  (i.e.  online  delivery 
instructional  technologies.)’  Three  students  did  not  give  any  response.  Below  are  the 
student  responses: 


Positive  reactions 

•  I  felt  the  media  used  to  present  the  instruction  was  quite  appropriate.  The 
mix  of  web  links  and  instructor  input  in  the  form  of  the  module  readings 
made  synthesis  of  the  text  material  much  easier  and  more  thorough. 

•  I  liked  them  uh-lot 

•  Favorable  reaction 

•  Good  when  it  worked 

•  I  thought  it  was  fine 

•  Interesting 

•  It  was  great  when  it  was  working.  I  am  not  sure  I  understand  this  question 

•  It  was  OK  when  it  worked 

•  It  works  for  me! 

•  Over  all  I  like  it 

•  Really  enjoyed  the  links  and  instructor  presentations  for  modules 

•  The  media  was  good  when  it  was  up 

•  The  media  was  good,  with  the  notable  exception  of  having  to  pick  up 
hardcopies  of  documents  from  the  instructor.  This  defeated  the  purpose  of 
online  instruction;  I  should  get  everything  from  the  computer. 
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Neutral  reactions  or  suggestions 


•  I  prefer  classroom 

•  Need  a  search  engine.  Plus  better  navigational  tools 

•  No  reaction. . .  Just  reality 

•  None. 

•  Thought  the  media  was  adequate  for  the  general  concepts  of  the  course.  I 
thought  that  if  the  course  were  to  get  more  technical,  a  different  media 
would  be  required 

Negative  or  skeptical  reactions 

•  The  technology  was  not  helpful.  Caused  several  delays  in  the  course  and 
was  not  always  user  friendly. 

•  Too  much  flexibility 

The  last  open-ended  question  simply  gave  students  opportunity  to  add  other 
comments.  Fifteen  of  them  did  not  have  any  additional  comments.  Those  who  decided  to 
comment  responded  as  follows: 


Positive  comments 

•  CDR  Higgins’  positive  energy  was  a  real  plus! 

•  Great  course  to  teach  on-line.  The  danger  here  is  that  because  one  course 
type  works  well  on-line,  we  start  trying  to  do  other  courses  on-line  that 
just  don’t  lend  themselves  well  to  the  venue,  like  heavy-math  type 
courses,  EO  courses,  etc. 

•  In  this  environment  I  would  prefer  a  classroom  environment 

Neutral  comments  or  suggestions 

•  Need  feedback  on  forum 

•  The  forum  should  be  real-time.  Schedule  multiple  meeting  times  within 
the  groups  to  have  online  chats.  Then  post  discussion  for  bulletin  board 
responses  from  others. 

•  I  do  not  offer  any  improvements,  I  believe  a  student  will  get  as  much  out 
of  the  course  as  they  put  into  it.  The  fact  that  this  was  a  distance  learning 
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format  did  not  have  a  negative  impact  on  my  view  of  the  course.  CDR 
Higgins  always  kept  students  informed  and  maintained  a  motivation  level 
that  is  required  with  an  on-line  course.  I  will  certainly  recommend  SS301 1 
and  CDR  Higgins  to  other  students.  Thanks  for  all  the  great  instruction! 

•  Did  you  view  the  flexibility  of  this  on-line  course  as  beneficial  to  your 
ability  to  succeed  in  this  class?  Yes  -  when  all  the  technical  problems  are 
sorted  out.  However,  due  to  this  being  for  first  time  the  amount  of 
technical  problems  removed  this  flexibility  as  site  was  often  down  when  it 
would  have  been  convenient  to  work  on  the  modules/get  the  assignment. 
As  a  result,  there  were  several  occasions  when  we  had  to  cram  to  complete 
the  assignments  when  it  was  not  a  convenient  time  to  work  on  it  because 
the  site  was  down  when  it  would  have  been  convenient. 

•  .  It's  a  good  course,  but  not  for  me  this  quarter 


The  last  two  categorical  but  non-ordered  variables  in  the  instrument  represented 
the  on-line  version  of  the  Long-Dziuban  Reactive  Behavior  Protocol:  a  set  of  four  boxes 
with  responses  was  used  to  measure  personal  on-line  behavior  types,  and  another  set  of 
four  boxes  with  responses  was  used  to  measure  personal  traits.  In  the  first  case,  the 
students  were  able  to  choose  only  one  box,  and  in  the  latter  case  they  could  choose  from 
none,  to  four  traits.  Figure  64  shows  distribution  of  the  responses  on  the  self-reported 
types.  In  our  sample,  the  Passive  Independent  type  dominated;  there  were  no  responses 
for  the  Passive  Dependent  type. 
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Long-Dziuban's  Reactive  Behavior  Protocol  - 
Personal  Types 
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Figure  64.  Distribution  of  the  Responses  to  Long-Dziuban’s  Reactive  Behavior 

Protocol  -  Personal  Types. 


This  distribution  is  surprising.  Dziuban,  Moskal,  and  Dziuban  (1999)  found  only 
12%  of  University  of  Central  Florida  (UCF)  students  attending  on-line  courses  (N=381) 
to  belong  to  Passive  Independent  (PI)  type,  while  in  this  study  eleven  students  or  50% 
belonged  to  the  PI  type.  Passive  Dependent  (PD)  was  the  least  frequent  type  in  both 
cases:  5%  in  Dziuban’ s  study  and  none  in  this  thesis  research.  Aggressive  Dependent 
(AD),  most  frequent  in  Dziuban’ s  study  (60%),  was  second  most  frequent  in  our  sample 
(six  students  or  27.27%),  while  23%  of  students  belonged  to  Aggressive  Independent 
type  in  Dziuban’s  study,  similar  to  this  study  (five  students  or  22.73%). 

The  sample  in  this  case  study  is  small,  and  the  students  were  not  university 
students  as  these  in  Dziuban’s  study  but  military  officers  selected  and  self-selected  for 
graduate  level  education  in  NPS.  There  is  not  enough  data  to  conclude  whether  this 
difference  is  caused  by  different  (highly  competitive)  environments,  of  differences 
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between  the  two  populations,  of  process  of  selection  of  NPS  students,  or  of  other  factors. 
If  Long’s  types  are  interpreted  as  stabile  personality  types,  the  difference  would  probably 
be  the  result  of  the  selection  and  self-selection  of  students,  but  because  Long  and  other 
authors  indicated  that  there  are  possibilities  of  influence  to  typical  reactive  behavior 
patterns  (for  instance,  by  counseling  them),  such  a  hypothesis  is  not  valid. 

Figures  65  and  66  show  responses  to  the  second  part  of  the  Protocol,  with  the 
combined  responses  shown  separated  in  the  first  graph,  and  included  in  the  basic  traits  in 
the  second  one.  Clearly,  compulsive  trait,  alone  (eight  students  or  42.11%)  or  combined 
with  impulsive  trait  (five  students  or  26.32%),  dominated.  Again,  interpretation  is 
difficult;  it  is  very  hard  to  distinguish  personal  preferences,  environmental  impacts,  and 
other  possible  factors  that  resulted  with  this  distribution. 


Figure  65.  Distribution  of  the  Responses  to  Long-Dziuban’s  Reactive  Behavior 
Protocol  -  Personal  Traits,  with  Separated  Combinations.13 


13  According  to  the  authors  of  the  instrument,  in  this  scale  a  subject  can  choose  any  combination  of 
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It  was  difficult  to  find  the  appropriate  statistical  method  to  check  the  level  of 
association  or  correlation  of  responses  to  this  Protocol  with  other  variables  in  the 
instruments.  I  decided  to  assign  1  to  the  responses  a  student  chose,  and  0  to  not-chosen 
responses,  and  I  calculated  correlations  with  other  variables,  using  only  coefficients  of 
rank  correlation  because  of  a  very  small  number  of  observations.  Table  64  shows  all  the 
statistically  significant  correlations.  Also,  in  these  calculations  I  used  responses  to  the 
second  part  of  the  Protocol  or  personal  traits  with  combinations  included  in  basic  four 
traits.  There  were  only  three  significant  correlations.  First  correlation  shows  that  PI  type 
was  associated  with  impulsive  trait  in  this  sample.  Other  correlations  are  very  hard  to 
interpret. 


Long-Dziu  ban's  Reactive  Behavior  Protocol  - 
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Figure  66.  Distribution  of  the  responses  to  Long-Dziuban’s  Reactive  Behavior 
Protocol  -  Personal  Traits,  with  Combinations  Included  into  Basic  Traits. 


responses,  from  none  to  all  four.  In  this  case  students  chose  from  none  to  two  traits  -  which  I  call 
‘combinations’. 
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Variables 

P 

p  value 

‘Passive  Independent  type’  and  ‘Impulsive  trait’ 

0.691 

<  0.001 

‘Aggressive  Dependent  type’  and  question 
number  10:  ‘How  useful  were  the  on-line  Module 
readings  in  helping  you  to  understand  the  course 
concepts,  objectives  and  principles?’ 

0.471 

0.023 

None  personal  traits  chosen  and  question  number 
24:  ‘Compare  the  amount  of  interaction  you  had  with  the 

instructor’ 

0.419 

0.047 

Table  64  Significant  Correlations  and  Levels  of  Significance  Between  Long- 
Dziuban’s  Reactive  Behavior  Protocol  Variables  (Personal  Traits  Include  Combinations 

in  Basic  Traits). 


Final  survey  -  Summary 

The  last  on-line  questionnaire  was  administered  anonymously.  Twenty-three  of 
thirty  students  filled  out  the  questionnaire.  The  questionnaire  consisted  of  thirty-one 
items,  including  five  open-ended  questions  or  statements. 

In  terms  of  student  views  on  their  final  status  in  achieving  the  course  learning 
objectives,  most  students  replied  positively.  Most  of  objectives,  according  to  distributions 
of  student  responses,  were  met.  Correlations  with  other  variables  were  significant  mostly 
within  this  same  group  of  variables  (asking  about  meeting  different  course  objectives), 
but  there  were  other  significant  correlations  too. 

The  second  group  of  questions  addressed  how  useful  the  students  viewed  some  of 
the  course  features:  guidelines  in  the  course  homepage,  guidelines  for  posting  to  the 
forums,  the  textbook,  the  module  on-line  readings,  linked  websites,  the  forum,  exercises, 
and  the  quizzes  as  feedback  to  the  students.  Most  distributions  strongly  tended  to  positive 
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responses,  but  there  were  significant  numbers  of  negative  responses  to  questions 
addressing  usefulness  of  forums  and  quizzes. 

The  third  set  of  questions  asked  students  how  much  they  liked  some  of  the 
features  like  forum  assignments,  textbook  reading  assignments,  the  module  readings,  and 
website  links.  Responses  on  usefulness  and  on  liking  the  same  features  (module  readings, 
forum  exercises/assignments)  correlated,  but  coefficients  were  not  high:  liking  and 
assessing  utility  of  same  feature  is  similar,  but  is  definitely  not  the  same. 

Responses  to  the  open-ended  questions  showed  some  trends.  Asked  what  other 
instructional  materials  should  be  included  in  the  course,  five  students  responded  ‘none’, 
but  eight  students  gave  one  or  more  suggestions.  Regarding  the  question  about  their 
reactions  to  the  instructional  strategies  used  in  the  course,  eleven  students  responded 
positively,  four  neutrally,  and  four  responded  negatively  or  with  critiques.  Asked  whether 
this  course  fundamentally  changed  their  approach  to  learning,  most  students  (sixteen) 
responded  negatively,  but  three  agreed,  and  four  partially  agreed.  Their  reactions  about 
on-line  delivery  of  instructional  technologies  were  mostly  positive  (thirteen  responses), 
some  neutral  (five),  and  only  two  negative.  For  additional  comments,  the  students 
provided  three  positive  points  of  view,  and  several  neutral  comments  and  suggestions. 

The  flexibility  of  the  on-line  course  in  the  sense  of  time  management,  mostly 
seemed  to  be  more  beneficial  to  the  students  success  in  other  classes/courses  than  in  this 
course. 

Asked  about  interactivity,  most  students  thought  that  the  amount  of  interactivity 

between  the  students  and  the  instructor  in  the  course  was  less  than  in  a  face-to-face 

course,  but  they  were  much  more  uncertain  when  asked  about  the  quality  of  that 
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interaction.  They  were  also  not  sure  about  the  amount  of  interaction  between  the  students 
and  about  the  quality,  but  the  majority  thought  that  the  quality  was  lower  in  an  on-line 
course. 

The  student  responses  to  Long-Dziuban’s  Reactive  Behavior  Protocol  were  very 
interesting,  showing  obvious  differences  from  findings  in  other  studies.  Because  of  a 
small  number  of  observations  and  some  other  methodological  limitations,  especially  in 
terms  of  selectivity  and  self-selectivity  that  formed  this  group  of  students  (military 
officers  under  pressure  of  competitiveness),  an  interpretation  of  these  results  is  difficult. 


B.  STUDENT  OPINION  FORMS  (SOFS) 

Twenty-two  students  completed  the  Student  Opinion  Forms  anonymously.  The 

first  part  of  the  SOFs  contains  data  on  which  curriculum  the  student  attends,  about  the 

number  of  credit  hours,  and  about  the  number  of  quarters  already  completed.  As  already 

stated,  all  of  the  students  in  the  sample  attended  the  same  curriculum.  Figure  67  shows 

the  distribution  of  the  numbers  of  hours  per  week  the  students  had  during  the  quarter. 


Distribution  of  Credit  Hours  in  the  Sample 
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Figure  67.  Weekly  Number  of  Credit  Hours  Distribution  in  the  Sample. 
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N  =  22 


Mean  =16.73 
Maximum  =  24 


Minimum  =  12 
Standard  deviation  =  2.90 


Most  of  the  students  in  the  sample  had  between  14  and  credit  19  hours  (17  of  22, 
or  77.27%),  but  the  range  between  the  minimum  and  maximum  is  relatively  large  -  a  12 
hour  difference.  The  calculation  of  the  rank  correlations  between  the  number  of  hours  and 
any  other  variable  in  SOFs  showed  no  significant  relationships.  In  other  words,  credit 
hours  did  not  make  any  difference  in  any  of  the  other  variables  in  the  instrument. 

The  second  variable  was  about  the  number  of  quarters  a  student  already 
completed  at  NPS  before  the  current  quarter.  Figure  68  shows  the  distribution  of  the 
responses  with  the  basic  statistical  indicators. 


Distribution  of  the  Number  of  Quarters  Completed 
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Figure  68.  Distribution  of  the  Number  of  Quarters  Completed. 

N  =  22  Mean  =  4.00  Minimum  =  0 

Maximum  =  6  Standard  deviation  =  1 .63 

The  relative  majority  of  the  students  (10  of  22,  or  over  45%)  were  in  their  sixth 

quarter  of  their  curriculum  during  the  course,  but  there  were  individual  students  with 
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none  to  six  quarters  finished.  Again,  there  were  no  significant  rank  correlations  with  any 
other  variable. 

Answering  the  next  question,  all  of  the  students  confirmed  that  the  course  was 
required  in  their  curriculum  and  was  not  an  elective. 

After  describing  basic  data,  the  sixteen  statements  asked  the  students  for  their 
opinions  about  the  course  and  the  instructor.  In  addition,  SOFs  provided  students  an 
opportunity  to  write  their  own  comments.  Responses  to  the  sixteen  statements  are  in  the 
form  of  a  five-point  scale.  For  the  first  eleven  statements,  the  scale  had  the  folllowing 
descriptions:  ‘strongly  agree’;  ‘agree’;  ‘no  strong  opinion’;  ‘disagree’;  ‘strongly 
disagree’;  and  ‘no  comment’.  The  latter  means  the  question  is  irrelevant  for  the  course  or 
the  student  does  not  want  to  respond  to  the  question.  In  the  last  five  statements,  where 
students  provided  an  overall  rating  of  the  instructor,  the  course,  the  textbook,  the  quality 
of  the  exams,  and  the  laboratories,  responses  were:  ‘outstanding  (among  the  top  10%)’; 
‘excellent  (among  the  top  30%)’;  about  average  (middle  40%)’;  ‘fair  (in  the  lowest 
30%)’;  and ‘poor  (in  the  lowest  10%)’. 

I  did  not  include  ‘No  comment’  or  ‘Not  applicable’  responses  in  the  graphs.  These 
are  only  mentioned  in  parentheses  because  they  do  not  represent  an  opinion  but  the 
unwillingness  for  subjective  reasons  to  give  an  opinion  about  that  statement.  Data  on 
median,  maximum,  minimum,  or  mode  category  of  the  responses  to  these  question  do  not 
include  ‘No  comment’  or  ‘Not  applicable’  category  responses.  The  same  holds  true  for 
rank  correlations.  For  this  reason,  the  higher  p  value  in  come  cases  may  be  less 
significant  because  of  the  smaller  number  of  observations  (N)  in  the  calculation. 
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Therefore  I  ordered  the  coefficients  of  rank  correlation  by  significance.  If  the  significance 
was  equal,  I  ordered  them  by  the  size  of  the  coefficient. 

The  sixteenth  statement  (‘Overall,  I  would  rate  the  laboratories’)  is  not  applicable 
for  this  course  and  is  therefore  disregarded. 

The  distribution  of  the  responses  to  the  first  statement  is  shown  in  Figure  69. 


'The  Course  Was  Well  Organized" 


Opinion 


Figure  69.  Responses  to  the  First  Statement  in  the  SOFs. 

N  =  22  Mean  =  3.59  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

From  the  distribution  in  Figure  69,  student  opinions  on  the  organization  of  the 
course  varied  from  very  negative  and  moderately  negative  (three  students  or  13.64%)  or 
neutral  (six  students  or  27.27%),  to  positive  responses,  which  is  the  majority  (thirteen 
students  or  59.09%).  In  general,  most  of  student  were  at  least  neutral  or,  predominantly, 
positive  about  course  organization. 
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The  responses  to  the  first  statement  (‘The  course  was  well  organized’)  have 
significant  correlations  with  twelve  of  the  other  fourteen  variables  (Table  65). 


Statement 

number 

Statement 

P 

p  value 

7 

‘The  instructor  was  prepared  for  the  course’ 

O.86OI4 

<0.001 

9 

‘The  instructor  made  this  course  a  worthwhile  learning 
experience’ 

0.762 

<0.001 

14 

‘Overall,  I  would  rate  the  textbook’ 

0.726 

<0.001 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.724 

<0.001 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.703 

<  0.001 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.702 

<  0.001 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.648 

<  0.001 

13 

‘Overall,  I  would  rate  this  course’ 

0.671 

0.001 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn 

0.654 

0.001 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’: 

0.649 

0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.644 

4 

‘Difficult  concepts  were  made  understandable’ 

0.598 

2 

‘Time  in  class  was  spent  effectively’ 

0.82515 

Table  65.  Correlations  and  Levels  of  Significance  between  Responses  to  the  First 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 

Their  responses  to  the  first  statement  significantly  correlated  to  the  responses  to 
almost  all  the  other  statements;  statements  that  represent  desirable  qualities  of  the  course, 
the  instructor,  or  the  textbook.  The  organization  of  the  course  is  very  important  and  is  a 
‘related-to-everything’  characteristic  within  the  context  of  the  course.  One  of  the  highest 
correlations  in  this  instrument  is  between  the  first  and  the  seventh  statement.  This 


14  After  elimination  of  ‘No  comment’  or  ‘Not  applicable’  responses,  number  of  pairs  of  responses  (N) 
for  calculation  of  this  correlation  was  18.  N  for  any  other  variable  in  this  set  is  22  minus  the  number  of  ‘No 
comment’  responses,  which  is  visible  in  the  graph  of  distribution  of  responses  to  that  variable. 

15  For  the  same  reason,  N  of  data  pairs  taken  into  account  was  only  nine. 
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correlation  means  that  the  students’  view  on  organization  of  the  course  and  on  the  quality 
of  the  instructor’s  preparation  highly  correlates. 

It  is  interesting  to  see  which  question  the  first  question  was  not  significantly 
correlated  with.  This  turned  out  to  be  the  third  question:  ‘The  instructor  seemed  to  know 
when  students  didn’t  understand  the  material’.  This  non-significance  of  the  correlation 
probably  appeared  because  of  the  changed  nature  of  the  feedback  from  the  instructor  to 
the  students  in  the  DL  course.  Feedback  from  the  instructor  to  students  in  DL  is  mostly 
individual;  therefore,  very  different  than  in  a  ‘normal’  course.  Crucial  difference  in 
comparison  with  a  face-to-face  class  is  that  media  used  for  the  instructor’s  feedback  in 
DL  course  (e-mail  and  other  similar  means)  excludes  the  instructors’  personal  presence. 
Therefore,  it  was  more  difficult  for  students  to  determine  if  the  instructor  was  able  to  see 
student  understanding  of  the  material. 

The  third  statement  was  ‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’  Figure  70  shows  the  following  distribution  of  responses. 

As  in  many  other  variables  in  this  set,  positive  opinions  (there  are  nine  such 
responses  or  50%)  dominate,  but  there  are  also  seven  neutral  and  one  negative  opinion. 
Correlations  are  in  Table  66.  They  are  overall  less  numerous,  smaller  in  size  and  with 
lower  significances  than  the  first  statement. 
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"The  Instructor  Seemed  to  Know  When  Students 
didn't  Understand  the  Material" 
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Figure  70.  Responses  to  the  Third  Statement  in  the  SOFs. 

N  =  1 8  Mean  =  3.39  Median  =  3 

Minimum  =  2  Maximum  =  5  Mode  =  3 


Statement 

number 

Statement 

P 

p  value 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.655 

0.006 

4 

‘Difficult  concepts  were  made  understandable’ 

0.583 

0.018 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area. 

0.559 

0.025 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.555 

0.026 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.541 

0.031 

9 

‘The  instructor  made  this  course  a  worthwhile  learning 
experience’ 

0.537 

0.032 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.525 

0.037 

6 

‘I  felt  free  to  ask  questions’ 

0.486 

0.041 

Table  66.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Third 

and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 
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The  next  statement  in  SOFs  was  the  following:  ‘Difficult  concepts  were  made 
understandable’.  Figure  71  shows  the  distribution  of  the  responses.  Only  one  student 
disagreed  (strongly)  with  the  statement,  and  one  chose  not  to  answer.  Again,  there  were  a 
number  of  neutral  opinions  (seven  or  33.33%),  but  positive  (‘agree’  -  nine  or  42.86%; 
‘strongly  agree’  -  four  or  19.05%)  dominated. 


Figure  7 1 .  Responses  to  the  Fourth  Statement  in  the  SOFs. 

N  =  21  Mean  =  3.71  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

This  variable  had  significant  correlations  with  almost  all  other  variables  in  the 
instrument  (Table  67).  The  highest  correlations  were  with  ‘overall  course  organizational 
indicators’:  overall  course  rating,  how  the  instructor  prepared  for  the  course,  and  the 
course  organization. 
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Statement 

number 

Statement 

P 

p  value 

13 

‘Overall,  I  would  rate  this  course’ 

0.766 

<  0.001 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.749 

<  0.001 

1 

‘The  course  was  well  organized’ 

0.677 

<  0.001 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’ 

0.653 

0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.689 

0.002 

9 

‘The  instructor  made  this  course  a  worthwhile  learning 
experience’ 

0.567 

0.007 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.578 

0.010 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.018 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.504 

0.020 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.486 

0.030 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 
subject’ 

0.481 

0.032 

14 

‘Overall,  I  would  rate  the  textbook’ 

0.467 

0.033 

Table  67.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Fourth 

and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 


The  fifth  question  (T  had  confidence  in  the  instructor’s  knowledge  of  the  subject’) 
had  almost  only  positive  student  responses:  fifteen  of  them  (71.43%)  chose  ‘strongly 
agree’  and  five  (23.81%)  ‘agree’.  Only  one  decided  to  give  a  neutral  response,  and 
another  chose  not  to  respond.  Figure  72  shows  the  distribution  of  the  responses,  and 
Table  68  lists  all  the  significant  correlations  with  other  variables  in  the  instrument.  There 
are  eleven  correlations  with  a  level  of  significance  under  0.05.  Interestingly,  the  highest 
correlation  is  with  responses  to  the  question  about  feeling  free  to  ask  the  instructor 
questions.  This  is  to  say,  the  more  students  had  confidence  in  the  instructor’s  knowledge 
and  expertise,  the  more  they  felt  free  to  ask  her  questions  within  the  course  context. 
Another  high  correlation  was  with  responses  to  the  first  question  about  the  overall 
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organization  of  the  course.  If  the  students  see  the  course  as  better  organized,  the  more 
they  felt  free  to  ask  the  instructor  questions. 


"I  Had  Confidence  in  the  Instructor's  Knowledge  of 

the  Subject" 


Opinion  (no  opinion:  1  student) 


Figure  72.  Responses  to  the  Fifth  Statement  in  the  SOFs. 


N  =  21 

Minimum  =  3 


Mean  =  4.67  Median  =  5 

Maximum  =  5  Mode  =  5 
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Statement 

number 

Statement 

p  value 

6 

‘I  felt  free  to  ask  questions’ 

0.800 

<  0.001 

1 

‘The  course  was  well  organized’ 

0.702 

<0.001 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.628 

0.002 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.606 

0.003 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’ 

0.548 

0.010 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.535 

0.012 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.550 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.487 

4 

‘Difficult  concepts  were  made  understandable’ 

0.481 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.459 

14 

‘Overall,  I  would  rate  the  textbook’ 

0.434 

Table  68.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Fifth 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 

I  already  mentioned  the  next  question  (‘I  felt  free  to  ask  questions’)  when 
commenting  on  the  previous  question.  All  the  responses  to  this  question  about  feeling 
free  to  ask  the  instructor  questions  during  the  course  were  positive  -  14  ‘strongly  agree’ 
and  8  ‘agree  (Figure  73).  The  issue  of  the  instructor’s  very  high  responsiveness  to  the 
students  was  mentioned  a  number  of  times  in  the  interviews.  Eleven  correlations  were 
significant,  and  four  correlations  had  p  0.001  or  less  (Table  69). 
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Figure  73.  Responses  to  the  Sixth  Statement  in  the  SOFs. 

N  =  22  Mean  =  4.64  Median  =  5 

Minimum  =  4  Maximum  =  5  Mode  =  5 


Statement 

number 

Statement 

p  value 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.729 

<0.001 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.719 

<0.001 

1 

‘The  course  was  well  organized’ 

0.644 

4 

‘Difficult  concepts  were  made  understandable’ 

0.689 

HT»M 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.659 

0.002 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.630 

0.002 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.628 

0.002 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’ 

0.579 

0.005 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.570 

0.006 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.486 

0.041 

Table  69.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Sixth 
and  to  Other  Statements  (Ordered  by  Level  of  Significance). 
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Responses  to  the  sixth  question  most  highly  correlated  with  responses  concerning 
the  students’  view  of  the  instructor’s  knowledge,  her  preparation  for  the  course,  her 
making  the  objectives  clear,  and  with  views  on  the  overall  organization  of  the  course. 

The  seventh  question  was  about  the  instructor’s  preparedness  for  the  course. 
Figure  74  shows  the  distribution  of  the  responses. 


Figure  74.  Responses  to  the  Seventh  Statement  in  the  SOFs. 

N  =  18  Mean  =  4.00  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 


Only  one  student  (5.56%)  responded  with  disagree  (strongly)  with  the  seventh 

statement.  Three  students  (16.67%)  chose  neutral  responses,  and  four  other  students 

(22.22%)  decided  not  to  respond  (‘No  comment’).  However,  correlations  (Table  70)  are 

numerous  and  some  very  high  (six  of  them  with  p  under  0.001).  It  could  be  concluded 

that  the  correlations  of  student  responses  to  all  other  variables  covered  in  the  SOFs  reflect 

very  important,  and  maybe  a  central  role  with  this  issue  within  the  overall  context  of  the 

course.  Students  clearly  think  that  the  instructor’s  preparedness  has  a  strong  connection 
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with  the  organization  of  the  course  (statement  1),  quality  of  exams  (statement  15), 
making  difficult  concepts  understandable  (statement  4),  freedom  to  ask  questions 
(statement  6),  and  overall  rating  of  the  course  (statement  13). 


Statement 

number 

Statement 

P 

1 

‘The  course  was  well  organized’ 

0.860 

<  0.001 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.830 

13 

‘Overall,  I  would  rate  this  course’ 

0.777 

<0.001 

4 

‘Difficult  concepts  were  made  understandable’ 

0.749 

<0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.729 

<0.001 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’ 

0.715 

<  0.001 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.677 

0.002 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.632 

14 

‘Overall,  I  would  rate  the  textbook’ 

0.562 

0.015 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.550 

0.018 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.555 

0.026 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.503 

0.033 

Table  70.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Seventh 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 

The  eighth  statement  was  ‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’.  Although  most  of  students  responded  with  ‘Strongly  agree’  (ten  of  them  or 
over  45%),  other  responses  were  dispersed  over  all  the  other  categories  (Figure  75).  Two 
students  (9.09%)  strongly  disagreed  and  one  disagreed.  Four  students  (18.18%)  chose 
neutral  responses.  In  summary,  more  student  responses  were  positive  (total  of  15  students 
or  68.1 8%)  than  neutral  or  negative  (seven  students  or  less  than  32%). 
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"The  Instructor's  Objectives  for  the  Course  Have 
Been  Made  Clear" 


Opinion 


Figure  75.  Responses  to  the  Eighth  Statement  in  the  SOFs. 

N  =  22  Mean  =  3.91  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  5 

Interestingly,  the  variables  having  the  highest  correlations  (Table  71)  with 
responses  to  this  statement  (statements  number  10,  9  and  1 1)  are  close  by  their  content  to 
the  instructor’s  approach  to  the  learning  process.  Through  my  contacts  and  interviews 
with  the  instructor  I  realized  she  invested  energy  to  make  the  course  very  stimulating  to 
students,  a  worthwhile  learning  experience,  and  she  cared  about  the  student  progress. 
More  a  student  understood  the  learning  objectives  of  the  course,  better  he  or  she  could 
understand  the  instructor’s  top  priorities,  what  these  correlations  confirm. 

It  would  be  interesting  to  examine  in  depth  intercorrelations  of  these  variables, 
but  the  data  (small  sample,  ordered  categorical  variables)  do  not  permit  more 
sophisticated  mathematical-analytical  methods  for  extracting  underlying  dimensions  and 
relationships. 
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Statement 

P 

p  value 

10 

‘The  instructor  stimulated  my  interest  in  the  subject 

area’ 

0.810 

<  0.001 

9 

‘The  instructor  made  this  course  a  worthwhile  learning 
experience’ 

0.752 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.745 

<0.001 

1 

‘The  course  was  well  organized’ 

0.724 

<0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.719 

<  0.001 

13 

‘Overall,  I  would  rate  the  course’ 

0.651 

0.001 

12 

‘Overall,  I  would  rate  this  course’ 

0.656 

0.002 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.662 

0.003 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.606 

0.003 

14 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.569 

0.006 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.528 

4 

‘Difficult  concepts  were  made  understandable’ 

0.507 

■HI 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.541 

0.031 

Table  7 1 .  Correlations  and  Levels  of  Significance  between  Responses  to  the  Eighth 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 


The  ninth  statement  (‘The  instructor  made  this  course  a  worthwhile  learning 
experience’)  also  produced  dispersed  student  responses,  with  ‘strongly  agree’  responses 
as  in  the  previous  question,  but  with  other  differences  in  distribution.  Three  students 
(13.64%)  disagreed  with  the  statement  (two  ‘strongly  disagreed’  and  one  ‘disagreed’)  and 
five  (22.73%)  chose  a  neutral  response  (Figure  76).  Significant  correlations  with 
respective  significance  levels  are  in  Table  72.  There  are  numerous  significant 
correlations,  and  six  of  them  have  p  equal  or  less  than  0.001 . 
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"The  Instructor  Made  this  Course  a  Worthwhile 
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Figure  76.  Responses  to  the  Ninth  Statement  in  the  SOFs. 

N  =  22  Mean  =  3.82  Median  =  4 


Minimum  =  1  Maximum  =  5  Mode  =  5 


Statement 

number 

Statement 

P 

p  value 

12 

‘The  instructor  cared  about  student  progress  and 
did  his  share  in  helping  us  to  learn’ 

0.833 

<0.001 

11 

‘The  instructor  stimulated  my  interest  in  the 
subject  area’ 

0.816 

<0.001 

1 

‘The  course  was  well  organized’ 

0.762 

<0.001 

8 

‘The  instructor’s  objectives  for  the  course  have 
been  made  clear’ 

0.752 

<0.001 

10 

‘The  instructor  made  the  course  a  worthwhile 
learning  experience’ 

0.691 

<  0.001 

13 

‘Overall,  I  would  rate  this  course’ 

0.672 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.677 

14 

‘Overall,  I  would  rate  the  textbook’ 

0.600 

6 

‘I  felt  free  to  ask  questions’ 

0.570 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of 
the  subject’ 

0.535 

0.012 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.528 

0.016 

3 

‘The  instructor  seemed  to  know  when  students 
didn’t  understand  the  material’ 

0.537 

0.032 

Table  72.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Ninth 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 
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The  question  whether  the  instructor  stimulated  students’  interest  in  the  subject 
area  was  next.  Two  students  (9.09%)  disagreed  (one  strongly),  and  six  students  (27.27%) 
chose  neutral  responses  (Figure  77). 


Figure  77.  Responses  to  the  Tenth  Statement  in  the  SOFs. 

N  =  22  Mean  =  3.95  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  5 

All  significant  correlations  with  other  variables  are  listed  in  Table  73.  They  were 

numerous  with  seven  of  them  having  p  equal  or  less  than  0.00 1 . 
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Statement 

number 

Statement 

P 

p  value 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.810 

<0.001 

13 

‘Overall,  I  would  rate  the  course’ 

7 

‘The  instructor  was  prepared  for  the  course’ 

9 

‘The  instructor  made  this  course  a  worthwhile  learning 
experience’ 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

4 

‘Difficult  concepts  were  made  understandable’ 

0.653 

0.001 

1 

‘The  course  was  well  organized’ 

0.649 

0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.005 

12  ' 

‘Overall,  I  would  rate  the  instructor’ 

0.007 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 
subject’ 

0.548 

0.010 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.559 

0.025 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.503 

0.027 

14 

‘Overall,  I  would  rate  this  textbook’ 

0.471 

0.027 

Table  73.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Tenth 

and  to  the  Other  Statements  (Ordered  by  Level  of  Significance) 


It  is  interesting  to  note  an  exceptionally  high  correlation  between  the  tenth  and 
eighth  statement.  In  the  students’  opinion,  the  clearer  the  instructor  made  course 
objectives,  the  more  stimulated  the  students  felt. 


The  eleventh  statement  in  the  Student  Opinion  Forms  was  the  following:  ‘The 
instructor  cared  about  student  progress  and  did  his  share  in  helping  us  to  leam’.  Figure  78 
shows  the  distribution  of  the  student  responses.  Nineteen  students  (86.36%)  agreed  with 
the  statement,  nine  of  them  ‘strongly’.  Only  three  students  (13.64%)  chose  neutral 
responses,  and  none  disagreed. 
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Figure  78.  Responses  to  the  Eleventh  Statement  in  the  SOFs. 

N  =  22  Mean  =  4.36  Median  =  4 

Minimum  =  3  Maximum  =  5  Mode  =  4 

Table  74  shows  significant  correlations  with  other  variables.  Eleven  correlations 
were  significant  with  p  under  0.05,  and  five  had  p  equal  or  less  than  0.001.  The  highest 
was  the  correlation  with  responses  about  the  instructor  making  the  course  a  worthwhile 
learning  experience. 
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Statement 

number 

Statement 

P 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.816 

<0.001 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.745 

<0.001 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.658 

<0.001 

10 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.679 

0.001 

1 

‘The  course  was  well  organized’ 

0.654 

0001 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.628 

0.002 

6 

‘I  felt  free  to  ask  questions’ 

0.628 

0.002 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.655 

0.006 

13 

‘Overall,  I  would  rate  the  course’ 

0.510 

0.015 

15 

‘Overall,  I  would  rate  the  textbook’ 

0.481 

0.032 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.503 

0.033 

Table  74.  Correlations  and  Levels  of  Significance  between  Responses  to  the 
Eleventh  and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 

Five  five-point  rating  scale  questions  followed  after  the  first  eleven  statements.  In 
them  the  students  rated  the  instructor,  the  course,  the  textbook,  and  the  quality  of  the 
exams.  The  last  scale  about  the  laboratories  was  disregarded.  There  were  no  special 
‘laboratory  activities’. 

The  first  question  rated  the  instructor  (Figure  79).  Ratings  were  very  positive. 
Seven  students  (33.33%)  rated  the  instructor  as  ‘outstanding’  and  nine  (42.86%)  as 
‘excellent’.  Only  one  student  (4.76%)  chose  ‘fair’,  one  did  not  rate  the  instructor,  and  the 
remaining  students  (four  of  them  or  19.05%)  chose  ‘about  average’.  Looking  at  the 
correlations  (Table  75),  we  can  conclude  that  in  their  rating  of  the  instructor,  the  students 
mostly  considered  if  they  thought  the  course  was  ‘a  worthwhile  learning  experience’. 
Nine  other  correlations  were  significant. 
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Figure  79.  Responses  to  the  Twelfth  Statement/Scale  in  the  SOFs. 


N  =  21 


Mean  =  4.05  Median  =  4 


Minimum  =  2 


Maximum  =  5  Mode  =  4 


Statement 

number 

Statement 

P 

p  value 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.833 

<0.001 

1 

‘The  course  was  well  organized’ 

0.648 

<  0.001 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  leam’ 

0.658 

0.001 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.656 

0.002 

6 

‘I  felt  free  to  ask  questions’ 

0.630 

0.002 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.632 

0.005 

10 

‘The  instructor  stimulated  my  interest  in  this  subject 

area’ 

0.570 

0.007 

13 

‘Overall,  I  would  rate  this  course’ 

14 

‘Overall,  I  would  rate  the  textbook; 

0.498 

0.022 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.487 

0.025 

Table  75.  Correlations  and  Levels  of  Significance  between  Responses  to  the  Twelfth 
and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 
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The  students’  view  of  the  course  visibly  differs  from  their  view  of  the  instructor. 


The  students  rated  the  course  less  favorably  than  they  rated  the  instructor  (Figure  80). 


Figure  80.  Responses  to  the  Thirteenth  Statement/Scale  in  the  SOFs. 


N  =  22  Mean  =  3.32  Median  =  3 

Minimum  =  1  Maximum  =  5  Mode  =  3 

Five  students  (27.73%)  rated  the  course  below  average,  while  only  one  student 
(4.55%)  rated  the  instructor  below  average.  Four  students  (18.18%)  rated  the  course  as 
‘outstanding’,  while  seven  students  (31.82%)  chose  the  same  response  rating  for  the 
instructor. 

A  possible  conclusion  from  this  data  is  that  there  are  some  undesirable  elements 

within  the  course  that  were  not  a  direct  consequence  of  the  instructor’s  approach, 

knowledge,  teaching  or  communication  skills.  An  additional  and/or  alternative 

explanation  could  be  that  the  student  rating  of  the  instructor  is  more  about  the  instructor’s 

personality,  and  rating  of  the  course  is  more  about  every  other  factor  relevant  to  the 
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course’s  success.  However,  significant  associations  among  all  aspects  of  the  course 
covered  with  SOFs  clearly  show  that  it  would  be  extremely  difficult  to  distinguish  the 
subtle  factors  students  take  into  consideration  when  assessing  course  quality.  Table  76 
shows  significant  correlations  of  the  student  course  rating  with  other  variables. 


Statement 

number 

Statement 

P 

p  value 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.777 

<  0.001 

4 

‘Difficult  concepts  were  made  understandable’ 

MEZm 

<  0.001 

1 

‘The  course  was  well  organized’ 

mmm 

<0.001 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.672 

0.001 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.651 

0.001 

10 

‘The  instructor  stimulated  my  interest  in  this  subject 

area’ 

0.633 

0.002 

14 

‘Overall,  I  would  rate  the  textbook; 

0.565 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.564 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.510 

0.015 

15 

‘Overall,  I  would  rate  the  quality  of  the  exams’ 

0.525 

6 

‘I  felt  free  to  ask  questions’ 

0.512 

■tUlBM 

Table  76.  Correlations  and  Levels  of  Significance  between  Responses  to  the 
Thirteenth  and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 


The  fourteenth  scale  asked  the  students  to  rate  the  textbook.  With  the  SS3011 
course,  one  main  textbook  was  used.  The  majority  of  the  students  (Figure  81)  rated  the 
textbook  as  ‘excellent’  (twelve  students  or  54.55%)  or  ‘outstanding’  (six  students  or 
27.27%),  but  two  students  (9.90%)  saw  the  textbook  as  ‘below  average’.  There  were  also 
two  (9.09%)  ‘about  average’  responses. 
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Figure  8 1 .  Responses  to  the  Fourteenth  Statement/Scale  in  the  SOFs. 

N  =  22  Mean  =  3.95  Median  =  4 

Minimum  =  1  Maximum  =  5  Mode  =  4 

Table  77  shows  significant  correlations  with  other  variables.  There  were  nine 

significant  correlations,  with  the  highest  correlation  with  the  first  statement  about  the 

organization  of  the  course. 


Statement 

number 

Statement 

P 

p  value 

1 

‘The  course  was  well  organized’ 

0.726 

<0.001 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.600 

6.003 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.569 

0.006 

13 

‘Overall,  I  would  rate  this  course’ 

0.565 

0.006 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.562 

0.015 

12 

‘Overall,  I  would  rate  the  instructor’ 

0.498 

0.022 

10 

‘The  instructor  stimulated  my  interest  in  this  subject 

area’ 

0.471 

0.027 

4 

‘Difficult  concepts  were  made  understandable’ 

0.467 

0.033 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 

subject’ 

0.434 

0.049 

Table  77.  Correlations  and  Levels  of  Significance  between  Responses  to  the 

Fourteenth  and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 
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The  last  scale  in  this  instrument  used  in  this  research  asked  the  students  about  the 
quality  of  the  exams  (Figure  82).  Responses  varied  from  ‘fair’  (three  or  15%)  to 
‘outstanding’  (two  or  10.00%),  with  ‘above  average’  responses  dominating  (eleven 
students  or  55.00%).  Two  students  did  not  choose  any  opinion. 

Eleven  correlations  were  significant  (Table  78).  The  highest  correlations  were 
with  the  student’s  view  of  the  instructor’s  preparedness  for  the  course  (seventh  statement) 
and  with  their  view  of  the  overall  organization  of  the  course  (first  statement). 


Figure  82.  Responses  to  the  Fifteenth  Statement/Scale  in  the  SOFs. 

N  =  20  Mean  =  3.50  Median  =  4 

Minimum  =  2  Maximum  =  5  Mode  =  4 
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Statement 

number 

Statement 

P 

p  value 

7 

‘The  instructor  was  prepared  for  the  course’ 

0.830 

<0.001 

1 

‘The  course  was  well  organized’ 

0.702 

<0.001 

6 

‘I  felt  free  to  ask  questions’ 

0.659 

0.002 

4 

‘Difficult  concepts  were  made  understandable’ 

0.578 

0.010 

' 

8 

‘The  instructor’s  objectives  for  the  course  have  been 
made  clear’ 

0.528 

0.014 

9 

‘The  instructor  made  the  course  a  worthwhile  learning 
experience’ 

0.529 

0.016 

13 

‘Overall,  I  would  rate  this  course’ 

0.525 

0.017 

10 

‘The  instructor  stimulated  my  interest  in  this  subject 

area’ 

0.503 

0.024 

11 

‘The  instructor  cared  about  student  progress  and  did 
his  share  in  helping  us  to  learn’ 

0.481 

0.032 

3 

‘The  instructor  seemed  to  know  when  students  didn’t 
understand  the  material’ 

0.525 

0.037 

5 

‘I  had  confidence  in  the  instructor’s  knowledge  of  the 
subject’ 

0.459 

0.042 

Table  78.  Correlations  and  Levels  of  Significance  between  Responses  to  the 
Fifteenth  and  to  the  Other  Statements  (Ordered  by  Level  of  Significance). 


SOFs  also  offer  students  a  possibility  to  add  their  own  comments.  Fifteen  students 
gave  comments.  After  analysis  of  their  content,  I  divided  the  comments  to  three  groups: 

Mostly  positive  opinions 

•  Overall,  this  was  an  excellent  course.  It  is  crucial  however  to  have  backup 
systems.  We  lost  our  rhythm  when  the  site  went  down. 

•  Started  out  as  excellent  class  -  web  site  being  down  (new  online  course  - 
somewhat  under  standing)  greatly  hampered  the  flexibility  of  the  course  -> 
which  is  supposed  to  be  one  of  the  benefits  for  an  online  course. 

•  I  appreciate  the  flexibility  of  the  DL  format.  Cdr  Higgins  did  an  excellent 
job  in  presenting  this  class  in  its  first  on-line  format.  I  learned  just  as  much 
or  more  than  I  would  have  learned  in  a  traditional  classroom  setting  and  I 
was  afforded  the  opportunity  to  do  the  work  as  my  schedule  allowed.  If 
given  the  opportunity,  I  would  choose  to  enroll  in  other  on-line  courses  as 
well. 
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Neutral  and  mixed  opinion 


•  #  316.  I  have  no  strong  opinion  on  this  question.  Instructor’s  gage  on 
student  progress  was  based  on  forum  input,  quiz  scores  and  evaluations.  If 
separate  contact  was  made  to  students  doing  poorly,  I  don’t  know. 

•  This  course  needs  more  time  to  develop  if  it  is  to  be  offered  on-line 
exclusively.  I  am  not  a  proponent  of  the  on-line  format  as  an  exclusive 
means  of  instruction  AT  a  university.  It  should  be  used  to  alignment 
classroom  instruction.  I  do,  however,  support  its  use  as  a  distance  learning 
tool. 

•  Took  the  on-line  version  of  course.  As  an  ITM  student,  I  was  just  looking 
for  a  god  overview  of  space  systems,  and  that's  what  the  course  delivered. 
Anyone  looking  for  MORE  than  an  overview  would  likely  not  get  what 
they  needed  in  the  online  version.  This  is  not  a  commentary  on  the 
instructor,  she  was  great,  more  a  commentary  on  not  being  able  to  interact 
one-on-one,  etc. 

•  Constant  small  group  interaction  is  the  key  to  staying  in  touch  with  the 
course  -  the  more  the  better.  I  did  not  do  my  part,  but  this  was  a 
worthwhile  learning  experience.  Spread  myself  too  thin  this  quarter;  this 
class  did  not  get  the  attention  that  it  deserves. 

•  First  iteration,  so  I  know  this,  but  need  to  insole  no  software  bugs  at  UCF. 
Also  recommend  tests  every  week.  While  I  appreciated  the  break,  it 
allowed  my  interaction  (especially  with  readings)  to  slide.  Overall  good 
experience. 

•  Cdr  Higgins  knows  the  material.  However  there  were  too  many  technical 
problems. 

•  I  feel  that  the  online  course  has  its  place  within  a  military  setting,  i.e.  on 
the  boat  or  you  are  just  taking  that  one  class.  There  is  no  reason  for  us  to 
take  on  on-line  course  while  we  are  here.  It  was  the  first  class  I  blew  off 
when  the  other  classes  got  busy.  I  need  the  classroom  interaction  in  order 
to  learn.  I  would  have  rather  sat  in  the  classroom  and  learned  something.  I 
feel  I  got  nothing  out  of  the  online  class. 

•  If  I  were  to  take  the  course  again,  I  would  take  the  in-class  version.  I 
enjoyed  the  material  but  felt  we  didn’t  get  enough  conversation  in  class. 

Negative  opinions  and  critiques 

•  Not  a  good  way  for  me  to  learn 


16  It  was  the  statement  ‘The  instructor  seemed  to  know  when  the  students  didn’t  understand  the 
material’. 
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•  Not  a  good  experience  at  all.  Really  was  a  waste  of  time.  Granted  it  tales  a 
lot  of  effort  on  both  the  instructor  and  student  parts.  The  technical 
problems  didn't  help,  but  it  seemed  as  if  we  were  going  week  to  week,  you 
can  only  talk  about  outsourcing  so  much.  The  information  covered  in 
modules  was  not  enough  to  properly  complete  the  final  point  paper. 

•  This  was  a  useless  class  in  terms  of  what  we  covered.  Modules  were  not 
ready,  they  were  difficult  to  access 

•  Tough  1st  go  around  for  online  course.  Many  glitches  in  the  system 
slowed  the  pace.  Need  Quizzes  on  each  module  just  to  keep  us  honest. 

SOFs  -  Summary 

I  made  several  conclusions  and  methodological  remarks  from  a  summary  of  the 
analysis  of  the  data  from  the  SOFs. 

The  responses  offered  in  the  SOFs  of  ‘No  comment’  or  ‘Not  applicable’ 
decreased  the  statistical  power  of  the  qualitative  analysis  because  such  responses  had  to 
be  eliminated  from  the  calculations. 

The  number  of  weekly  credit  horns  (sample  average:  16.73,  minimum  12, 
maximum  24)  and  the  number  of  completed  quarters  (sample  average:  4.00,  minimum  0, 
maximum  6)  do  not  have  any  significant  correlation  to  other  variables  in  the  set. 

The  second  statement  in  the  SOFs  (‘Time  in  class  was  spent  effectively’)  is  not 
applicable  in  DL  environments.  It  should  be  revised  or  removed. 

The  third  statement  in  the  SOFs  (‘The  instructor  seemed  to'  know  when  students 
didn’t  understand  the  material’)  also  seems  not  to  be  quite  appropriate  for  a  DL 
environment.  Its  meaning  changes  for  DL  courses  because  of  the  different  nature  of  the 
feedback  from  the  instructor. 
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In  general,  most  of  the  responses  to  the  different  statements  in  the  SOFs 
significantly  correlate.  Most  significant  correlations  have  statements  number  one  (‘The 
course  was  well  organized’),  seven  (‘The  instructor  was  prepared  for  the  course’),  eight 
(‘The  instructor’s  objectives  for  the  course  have  been  made  clear’),  nine  (‘The  instructor 
made  the  course  a  worthwhile  learning  experience’),  and  four  (‘Difficult  concepts  were 
made  understandable’).  Statement  number  two  (‘Time  in  class  was  spent  effectively’)  has 
the  lowest  number  of  significant  correlations  because  of  the  problem  stated  previously. 

Most  of  the  responses  were  very  favorable  about  the  instructor  and  the  course, 
especially  the  responses  to  statement  number  five  (‘I  had  confidence  in  the  instructor’s 
knowledge  of  the  subject’),  six  (‘I  felt  free  to  ask  questions’),  and  eleven  (‘The  instructor 
cared  about  student  progress  and  did  his  share  in  helping  us  to  learn’).  To  help  the  reader 
in  reviewing  all  responses  in  SOFs,  I  provide  summary  table  of  the  student  responses 
(Table  79)  at  the  end  of  this  summary. 

Although  predominantly  positive,  responses  to  the  majority  (11  of  15)  of 
statements  contained  some  individual  negative  responses  too. 

Some  individual  very  high  correlations  provide  interesting  insights  into  relations 
within  the  DL  course  context.  For  instance,  in  the  student’s  opinion,  the  more  the 
instructor  made  the  course  objectives  clear,  the  more  their  interest  in  the  subject  area  was 
stimulated.  However,  the  sample  size  and  kind  of  data  available  (ordered  categories)  do 
not  allow  for  more  subtle  analysis. 
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Frequencies  of  Responses 


Statements 

(N  =  22) 

Strongly  disagree 
/  Poor17 

Disagree  /  Fair 

No  strong  opinion 
/  About  average 

Agree  /  Excellent 

Strongly  agree  / 
Outstanding 

Not  applicable  / 
No  comment 

The  course  was  well  organized. 

2 

1 

6 

8 

5 

0 

Time  in  class  was  spent  effectively. 

0 

0 

5 

3 

13 

The  instructor  seemed  to  know  when 
students  didn’t  understand  the  material. 

0 

3 

6 

2 

D 

Difficult  concepts  were  made 
understandable. 

1 

0 

9 

DD 

I  had  confidence  in  the  instructor’s 
knowledge  of  the  subject. 

0 

0 

1 

5 

15 

1 

I  felt  free  to  ask  questions. 

0 

0 

0 

8 

15 

0 

The  instructor  was  prepared  for  the 
course. 

1 

0 

3 

8 

6 

D 

The  instructor’s  objectives  for  the 
course  have  been  made  clear. 

2 

1 

D 

5 

10 

0 

The  instructor  made  this  course  a 
worthwhile  learning  experience. 

2 

1 

5 

5 

9 

The  instructor  stimulated  my  interest  in 
the  subject  area. 

1 

1 

6 

DDD 

The  instructor  cared  about  student 
progress  and  did  her  share  in  helping  us 
to  learn. 

0 

0 

3 

10 

9 

0 

Overall,  I  would  rate  the  instructor. 

0 

1 

a 

9 

D 

1 

Overall,  I  would  rate  this  course. 

2 

3 

El 

6 

D 

0 

Overall,  I  would  rate  the  textbook. 

1 

1 

2 

12 

6 

0 

Overall,  I  would  rate  the  quality  of  the 
exams. 

2 

i 

3 

6 

9 

2 

0 

Table  79.  Summary  of  Student  Responses  in  Student  Opinion  Forms. 


17  Second  responses  (‘Poor’,  ‘Fair’,  etc.)  apply  for  last  four  statements  in  SOF. 
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C.  STUDENT  ON-LINE  TRACKING 

WebCT  software  used  in  the  course  was  able  to  automatically  track  the  student 

online  activities  during  the  course.  I  included  only  data  for  the  thirty  students  who 
completed  the  course  in  this  analysis.  Indicators  that  were  interesting  for  this  thesis  are: 

•  Total  number  of  hits,  i.e.  number  of  times  a  student  entered  the  course  web 
pages  using  his/her  logon  and  password 

•  Number  of  visits  to  the  course  homepage 

•  Number  of  visits  to  the  course  content  page 

•  Number  of  forum  articles  read,  i.e.  how  many  articles  in  all  of  forums 
together  a  student  read 

•  Number  of  forum  articles  posted  by  a  student 

•  Number  of  original  forum  articles  posted 

•  Number  of  follow-up  forum  articles  posted 

•  Number  of  different  course  pages  visited 

The  first  indicator,  total  number  of  hits,  was  widely  distributed,  from  a  minimum 
of  46  to  a  maximum  of  605  hits  by  one  student.  Figure  84  shows  the  distribution.  Most 
students  (fifteen  or  50.00%)  made  between  200  and  400  hits  during  the  course,  but  nine 
students  (30.00%)  had  over  500  hits.  Interestingly,  there  were  two  students  with  less  than 
100  hits,  but  they  completed  the  course  also. 

Since  the  student  tracking  was  recorded  by  student  names,  it  was  possible  to 
check  not  only  correlations  between  the  student  tracking  variables,  but  also  with  other 
instruments  and  variables  associated  with  student  names,  which  includes  the  First 
questionnaire,  the  Module  14  questionnaire,  and  final  grades.  In  cases  where  both 
variables  are  numeric  and  not  ordered  categories,  Pearson’s  r  correlation  coefficient  was 
calculated  with  the  respective  p  value,  not  Spearman’s  coefficient  of  rank  correlations. 
The  total  number  of  hits  had  nine  statistically  significant  correlations  (Table  80). 
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Total  Number  of  Hits 
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Number  of  hits  -  grouped 


Figure  83 .  Distribution  of  the  Total  Number  of  Hits. 

N  =  30  Mean  =  382.93  Minimum  =  46 

Maximum  =  605  Standard  dev.  =  149.97 


Instrument 

Variable  /  Question 

. Pfr 

p  value 

Student 

tracking 

Number  of  forum  articles  read. 

r  =  0.711 

<  0.001 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

p  =  0.581 

0.001 

Student 

tracking 

Number  of  visits  to  the  course  content  page. 

r  =  0.512 

0.004 

Student 

tracking 

Number  of  visits  to  the  course  homepage. 

r  =  0.463 

0.010 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  posting  comments  to  the  Forums?’ 

p  =  0.411 

0.015 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  accessing  the  Internet?’ 

p  =  0.425 

0.019 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.404 

0.021 

Student 

tracking 

Number  of  different  pages  visited. 

r  =  0.395 

0.031 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  sending  electronic  mail?’ 

p  =  0.387 

0.034 

Table  80.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 

Number  of  Hits  and  Other  Variables  (Ordered  by  Level  of  Significance). 
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From  this  table  we  can  see  that  students  who  accessed  the  on-line  course  materials 


strongly  tended  to  read  a  larger  number  of  forum  articles.  Also,  they  felt  more 
comfortable  with  quizzes,  with  posting  to  forums,  accessing  the  Internet,  and  sending  e- 
mail.  Also,  they  tended  to  visit  more  different  pages  in  the  on-line  course  material. 

The  second  student  tracking  indicator  was  the  number  of  visits  to  the  course 
homepage.  The  mean  of  this  variable  is  much  lower  than  the  mean  of  the  total  number  of 
hits  because  it  was  possible  to  use  on-line  course  materials  without  visiting  the  course 
homepage  by  using  favorites  or  bookmarks  in  the  Internet  browser  software.  Another 
possibility  is,  in  the  case  of  a  lost  connection  to  the  Internet  (e.g.,  the  TCP-IP 
connection),  to  logon  again  without  reopening  the  homepage. 

Figure  84  shows  the  distribution  of  the  visits  to  the  course  homepage.  There  were 
fewer  and  fewer  students  as  number  of  homepage  visits  increase  (75-104  visits  -  eight 
students  or  26.67%;  105-134  visits  -  six  students  or  20.00%;  135-164  visits  -  four 
students  or  13.33%),  but  there  were  also  two  students  (6.67%)  with  a  very  low  number  of 
visits.  This  indicator  had  significant  correlations  with  eleven  other  variables  (Table  81): 
six  other  student  tracking  indicators,  comfort  levels  with  accessing  the  Internet,  posting  to 
forums,  sending  e-mail,  and  even  the  number  of  quarters  enrolled.  The  latter  have  a 
negative  sign,  which  means  that  more  senior  NPS  students  tended  to  visit  the  course 
homepage  less,  which  might  mean  that  they  were  more  able  to  use  course  materials 
without  starting  from  the  course  homepage.  There  are  no  clear  indicators  to  support  this 
explanation. 
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Number  of  Visits  to  the  Course  Homepage 


Number  of  visits  •  grouped 


Figure  84.  Distribution  of  Visits  to  the  Course  Homepage. 


N  =  30 


Mean  =  90.80  Minimum  =15 

Maximum  =160  Standard  dev.  =  35.69 


Instrument 

Variable  /  Question 

p/r 

p  value 

Number  of  original  posts  to  the  forums. 

r  =  0.619 

<0.001 

Student 

tracking 

Number  of  visits  to  the  course  content  page. 

r  =  0.553 

0.002 

Student 

tracking 

Number  of  different  pages  visited. 

r  =  0.514 

0.004 

Student 

tracking 

Number  of  articles  posted  to  the  forums. 

r  =  0.508 

0.004 

First 

questionnaire 

Total  number  of  quarters  enrolled 

r  = -0.479 

0.007 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  accessing  the  Internet?’ 

p  =  0.475 

0.008 

Student 

tracking 

Total  number  of  hits. 

r  =  0.463 

0.010 

Student 

tracking 

Number  of  the  forums  articles  read. 

r  =  0.442 

0.015 

Module  14 

6  -  ‘Evaluating  the  likelihood  that  various  types  of 
space  countermeasures  might  be  used  across  the 
peace-war  continuum.’ 

p  =  0.408 

0.025 

Module  14 

1 0  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  posting  comments  to  the  Forums?’ 

p  =  0.406 

0.026 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at  this 
point  with  sending  electronic  mail?’ 

p  =  0.400 

0.029 

Table  8 1 .  Significant  Correlations  and  Levels  of  Significance  between  the  Total 

Number  of  Visits  to  the  Course  Homepage  and  Other  Variables  (Ordered  by  Level  of 

Significance). 
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The  number  of  visits  to  the  course  content  page  had  almost  in  shape  of  a  normal 
distribution  (Figure  85).  This  variable  had  only  three  significant  correlations  (Table  82). 
Only  correlations  with  other  student  tracking  indicators  were  significant. 


Figure  85.  Distribution  of  Visits  to  the  Course  Content  Page. 


N  =  30 


Mean  =  56.87  Minimum  =  16 

Maximum  =  99  Standard  dev.  =  21 .33 


Instrument 

Variable 

r 

p  value 

Student 

tracking 

Number  of  different  pages  visited. 

r  =  0.716 

<  0.001 

Student 

tracking 

Number  of  visits  to  the  course  homepage. 

r=  0.553 

Total  number  of  hits. 

r  =  0.512 

0.004 

Table  82.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 
Number  of  Different  Pages  Visited  and  Other  Variables  (Ordered  by  Level  of 

Significance). 


198 


The  number  of  forum  articles  read  (accessed)  had  an  extremely  wide  range  -  from 
only  eight  up  to  423  forum  articles  (Figure  87).  The  total  number  of  articles  in  the  course 
forums  was  585,  but  34  of  them  were  not  available  to  the  students.  Only  the  course 
management  staff  and  the  instructor  could  access  them.  Also,  when  the  instructor  broke 
forum  exercises  into  smaller  groups,  only  members  of  assigned  groups  and  the  instructor 
could  access  their  respective  forums.  In  other  words,  a  maximum  of  423  accessed  forum 
articles  is  very  close  to  a  theoretical  maximum  a  student  could  access  during  the  course. 

Eight  correlations  were  significant  (Table  83),  and  they  include  two  other  student 
tracking  indicators,  five  comfort  levels  from  the  Module  14  questionnaire  (word 
processing,  quiz,  accessing  the  Internet,  sending  e-mail,  and  posting  to  Forums),  and  self- 
perceived  computer  proficiency  from  the  First  questionnaire. 

Number  of  Forum  Articles  Accessed 

12 

10 

O 

Number  of  ® 
students  4 

2 
0 

N0°  oO°  nfP  *0° 

Number  of  articles  read  - 
grouped 

Figure  86.  Distribution  of  the  Number  of  Forum  Articles  Read. 

N  =  30  Mean  =  239.20  Minimum  =  8 

Maximum  =  423  Standard  dev.  =  1 16.09 
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The  next  item  was  the  total  number  of  articles  posted  to  the  forums.  The  total 
number  can  be  divided  into  original  posts  and  follow-up  posts,  which  cause  two  other 
indicators  to  be  used  in  this  data  analysis.  Figure  87  shows  the  distribution  of  the 
numbers  of  posted  articles.  On  average,  a  student  posted  approximately  14  to  15  articles. 
Most  of  the  students  posted  between  5  and  20,  but  the  total  range  is  from  one  to  31 
articles. 


Instrument 

Variable  /  Question 

p/r 

p  value 

Student 

tracking 

Total  number  of  hits. 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.478 

0.008 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

p  =  0.466 

0.009 

Module  14 

8  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  accessing  the  Internet?’ 

p  =  0.461 

0.010 

Student 

tracking 

Number  of  visits  to  the  course  homepage 

r  =  0.442 

0.015 

First 

questionnaire 

‘How  would  you  rate  your  current  proficiency  in 
using  computers’” 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  sending  electronic  mail?’ 

p  =  0.420 

0.021 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  posting  comments  to  the  Forums?’ 

p  =  0.373 

0.043 

Table  83.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 
Number  of  Forum  Articles  Readnnd  Other  Variables  (Ordered  by  Level  of  Significance). 

Table  84  shows  10  significant  correlations  with  other  variables:  the  highest 
correlation  found  in  this  thesis  research  is  between  the  number  of  posted  articles  and  the 
number  of  follow-up  articles  posted  to  the  forums.  Those  students  who  were  sending 
more  articles  regularly  posted  more  responses  to  others’  articles  in  the  forums. 
Correlations  with  comfort  levels  are  also  significant  with  word  processing,  posting  to  the 
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forums,  and  sending  e-mail.  Two  other  interesting  correlations  were  with  the  number  of 
quarters  enrolled,  with  senior  NPS  students  posting  less  articles,  and  with  final  grades  - 
students  more  active  in  posting  to  the  forums  tended  to  get  higher  final  grades. 


Number  of  Forum  Articles  Posted 


Number  of 
students 


Number  of  articles  posted  -  grouped 


Figure  87. 
N  =  30 


Distribution  of  Forum  Articles  Posted. 

Mean  =  14.53  Minimum  =  1 

Maximum  =  31  Standard  dev.  =  7. 1 1 
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Instrument 

Variable  /  Question 

p/r 

P  value 

Student 

tracking 

Number  of  follow-up  posts  to  the  forums. 

r  =  0.942 

<0.001 

Student 

tracking 

Number  of  original  posts  to  the  forums. 

r  =0.656 

<  0.001 

Student 

tracking 

Number  of  visits  to  the  course  homepage. 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.501 

0.005 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

p  =  0.497 

0.005 

Number  of  quarters  enrolled 

r  = -0.432 

0.017 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  posting  comments  to  the  Forums?’ 

p  =  0.416 

0.022 

Module  14 

3  -  ‘Interacting  with  a  subject-matter  expert 
whose  knowledge  you  can  use  to  solve  space- 
system  problems.’ 

p  =  0.404 

0.027 

Module  14 

9  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  sending  electronic  mail?’ 

p  =  0.398 

0.030 

FINAL  GRADE 

r  =  0.387 

0.035 

Table  84.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 

Number  of  Forum  Articles  Posted  and  Other  Variables  (Ordered  by  Level  of 

Significance). 

The  next  indicator  compared  the  number  of  original  posts  with  follow-up  posts. 
Follow-up  posts  are  forum  articles  written  as  reactions  to  posts/articles  written  by  other 
students  or  the  instructor,  what  is  graphically  visible  in  the  forums.  Figure  88  shows  the 
distribution.  Most  students  (twenty-five  or  83.33%)  posted  four  to  fifteen  follow-up 
posts.  Three  students  (10%)  posted  three  such  posts  or  less,  and  two  students  (6.67%) 
posted  over  20  follow-up  posts. 

Table  85  shows  significant  correlations.  There  are  seven  of  them.  This  variable 
correlates  with  final  grades  to  a  higher  degree  than  the  overall  number  of  posts  to  the 
forums.  Also,  the  correlation  between  the  total  number  of  posts  and  follow-up  posts  is 


higher  than  between  the  total  number  of  posts  and  original  posts.  The  conclusion  derived 
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from  these  relationships  is  that  the  instructor  valued  originality  in  the  student  posting 
forums,  what  had  an  impact  on  student  final  grades.  Three  correlations  with  comfort  level 
scales  from  the  Module  14  questionnaires  were  statistically  significant. 


Number  of  Original  Posts  to  the  Forums 


Number  of  posts  -  grouped 


Figure  88.  Distribution  of  Original  Posts  to  the  Forums. 
N  =  30  Mean  =  6.13  Minimum  =  1 


Maximum  =  13  Standard  dev.  =  13.17 


Instrument 

Variable  /  Question 

p  /  r 

P  value 

Student 

tracking 

Number  of  articles  posted  to  the  forums. 

Student 

tracking 

Number  of  visits  to  the  course  homepage. 

FINAL  GRADE 

Module  14 

10  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  posting  comments  to  the  Forums?’ 

p  =  0.485 

0.007 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

1 

p  =  0.427 

0.019 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.412 

0.024 

Student 

fracking 

Number  of  follow-up  posts  to  the  forums 

r  =  0.378 

0.040 

Table  85.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 
Number  of  Original  Articles  Posted  to  the  Forums  and  Other  Variables  (Ordered  by 

Level  of  Significance). 
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As  already  mentioned  above,  the  next  variable,  the  number  of  follow-up  posts  to 
the  forums  (Figure  89)  very  highly  correlates  with  the  total  number  of  forum  posts.  This 
variable  also  correlates  with  the  final  grades  (compare  Table  85  to  Table  84). 

Number  of  Follow-up  Posts  to  the  Forums 

10 
8 

Number  of  ® 
students  4 

2 

0 

0 

Figure  89.  Distribution  of  Follow-up  Posts  to  the  Forums. 

N  =  30  Mean  =  9.37  Minimum  =  0 

Maximum  =  26  Standard  dev.  =  2.14 
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Instrument 

Variable  /  Question 

p/r 

p  value 

Student 

tracking 

Number  of  follow-up  posts  to  the  forums 

r  =  0.942 

<0.001 

Module  14 

3  -  ‘Interacting  with  a  subject-matter  expert 
whose  knowledge  you  can  use  to  solve  space- 
system  problems.’ 

p  =  0.414 

0.023 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

p  =  0.414 

0.024 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.407 

0.026 

First 

questionnaire 

Number  of  quarters  enrolled. 

r  = -0.388 

0.034 

FINAL  GRADE 

r  =  0.379 

0.039 

Student 

tracking 

Number  of  original  posts  to  the  forums. 

r  =  0.378 

0.040 

Module  14 

1  -  ‘Identifying  key  elements  of  Space  Control.’ 

p  =  0.367 

0.046 

Table  86.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 
Number  of  Follow-up  Posts  to  the  Forums  and  Other  Variables  (Ordered  by  Level  of 

Significance). 


The  number  of  different  pages  visited  was  the  last  of  the  student  tracking 
indicators  (Figure  90).  Most  students  (twenty-two  or  73.33%)  accessed  all  the  available 
pages,  but  some  visited  fewer  pages:  five  students  (16.67%)  visited  14  to  15  pages;  one 
(3.33%)  visited  between  12  and  13,  one  between  10  and  11,  and  one  only  eight  web 
pages.  There  were  only  four  significant  correlations  (Table  87),  but  one  of  them  is  with 
final  grades.  Also,  self-perceived  skills  in  on-line  searching  correlate  significantly  with 
this  variable:  students  with  better  on-line  searching  skills  found  all  the  relevant  course 
pages  easier. 
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Number  of  Different  Pages  Visited 


Number  of  pages  -  grouped 


Figure  90.  Distribution  of  Numbers  of  Different  Course  Pages  Visited. 


N  =  30 


Mean  =15.8  Minimum  =  8 

Maximum  =17  Standard  dev.  =  2.14 


Instrument 

Variable  /  Question 

p/r 

Student 

tracking 

Number  of  visits  to  the  course  content  page. 

r  =  0.716 

<0.001 

FINAL  GRADE 

Student 

tracking 

Total  number  of  hits. 

r  =  0.395 

0.010 

First 

questionnaire 

‘How  would  you  rate  your  on-line  search 
resources  and  techniques?’ 

p  =  -0.404 

0.027 

Table  87.  Significant  Correlations  and  Levels  of  Significance  between  the  Total 
Number  of  Different  Number  of  Pages  Visited  and  Other  Variables  (Ordered  by  Level  of 

Significance). 


Besides  all  the  listed  data  from  the  automated  student  on-line  tracking,  I  also  used 
data  on  the  date  and  time  of  all  posts  in  all  forums  to  calculate  the  number  of  postings  by 
by  time  during  the  day,  and  by  day  in  a  week  (Figures  91  and  92).  This  data  provides 
additional  indicators  of  students’  on-line  activities  that  could  help  understanding  the 
overall  on-line  dynamics  within  the  course. 
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Percentage  of  Forum  Posts  by  Days  in  Week 


Day 


Figure  91 .  Distribution  of  the  Posts  by  Days  in  the  Week  (in  Percent)18. 

The  regular  time  to  switch  to  new  modules  during  the  course  was  on  Thursdays. 
Figure  91  suggests  that  the  peak  of  student  forum  postings  were  on  Thursdays  and  around 
Thursdays.  Thursday  was  a  usual  day  in  a  week  when  the  instructor  would  make  a  new 
module  available  to  the  students  and  expect  all  assignment  from  the  previous  module  to 
be  completed.  Another  weekly  peak  was  on  Mondays.  Students  used  the  convenience  of 
the  DL  course  by  posting  comments  on  Saturdays  and  Sundays. 

Figure  92  shows  the  average  number  of  forums  posts  by  time  of  day  for  all 
students  and  others  who  participated  in  the  course  forums.  A  24-hour  period  is  divided 
into  12  two-hours  parts.  There  are  three  peaks  of  forum  posting:  from  9AM  to  1PM,  from 
3PM  to  5PM,  and  from  9PM  to  11PM.  By  far,  the  fewest  number  of  posts  were  sent 
between  1AM  and  5AM.  Rarely  did  registered  course  students  access  the  on-line  course 

18  Note:  this  distribution  includes  all  posts  to  the  course  forums,  what  mean  not  only  those  by  students, 
but  by  the  instructor,  technical  support  staff  and  others. 
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materials  from  another  time  zone.  Therefore,  this  distribution  very  realistically  illustrates 
the  circadian19  aspect  of  this  DL  course:  NPS  students  in  the  sample  were  active  from 
morning  to  late  evening,  without  indications  in  this  data  set  that  they  often  take  an 
‘afternoon  nap’.  Moreover,  one  of  peaks  of  their  activity  was  very  late,  between  9  and  1 1 
PM. 


Figure  92.  Distribution  of  the  Forum  Posts  by  Time  of  Day  (in  percent)20. 


Student  Tracking  -  Summary 

All  eight  on-line  student  tracking  indicators  appeared  to  be  relevant  to  this  thesis 
They  all  correlated  significantly  with  between  three  and  eleven  other  variables  from  the 
First  questionnaire  (number  of  quarters  enrolled),  the  Module  14  questionnaire  (comfort 
level  in  on-line  activities  and  in  features  used  in  the  course)  or  with  final  grades  (what 
makes  them  relevant  as  secondary  indicators  of  student  achievement).  With  some 


19  There  are  numerous  studies  about  circadian  nature  of  human  activities,  for  instance  individual 
characteristics  related  with  work  in  day  or  night  shifts. 

20  Same  remark  holds  for  this  distribution. 


methodological  reserve,  all  of  them  have  notable  correlations  with  final  grades.  Analysis 
of  their  mutual  correlations  resulted  on  several  conclusions.  For  instance,  the  instructor 
valued,  in  terms  of  final  grades,  originality  in  the  student  postings  to  the  forums  more 
than  reactive  postings. 

Also,  almost  all  of  the  student  tracking  indicators,  except  two,  had  between  two 
and  five  significant  correlations  with  comfort  level  questions  in  the  Module  14 
instrument:  comfort  with  the  quiz  feature,  posting  to  forums,  accessing  the  Internet, 
sending  e-mail,  and  word  processing. 

Additional  insight  into  the  distribution  of  the  forum  postings  by  day  of  week  and 
by  time  of  day  illustrated  the  convenience  and  time  flexibility  the  students  had  in 
participating  in  the  forums. 


D.  FINAL  GRADES 

Final  grades  were  also  available- for  this  research.  To  convert  them  into  numerical 
variables,  I  assigned  points  to  the  grades  according  to  NPS  norm:  4.0  points  to  A,  3.7 
points  to  A-,  3.3  points  to  B+,  etc.  Figure  93  shows  the  distribution  of  the  final  grades. 
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Final  Grades  in  the  Course 


14 
12 
10 

Number  of  8 
students  6 
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A  A-  B+  B  B-  C+  C 
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Figure  93.  Distribution  of  the  Final  Grades  in  the  Course  (Grades  Converted  to 

Points:  A  =  4,  A-  =  3.7,  B+  =  3.3,  B  =  3.0...). 

N  =  30  Mean  =  3.71  Minimum  =  2.1 

Maximum  =  4  Standard  dev.  =  0.43 

Before  listing  the  significant  correlations  this  important  variable  had  with  other 

variables,  I  have  to  add  an  important  methodological  explanation:  some  of  variables 

correlating  significantly  with  final  grades  are  numerical,  so  that  Pearson’s  r  coefficient  of 

correlation  could  be  calculated.  However,  because  the  grades  are  distributed  very 

differently  from  a  normal  distribution  and  because  of  the  small  sample  size,  in  these  cases 

I  also  provide  p  coefficients  also.  The  validity  of  the  coefficients  of  rank  correlation  does 

not  depend  on  the  distribution  or  on  the  size  of  a  sample,  so  it  is  advisable  to  use  it  in  this 

case.  Correlations  are  listed  in  Table  88.  Note,  however,  that  in  cases  when  r  or  p  for  a 

numerical  variable  (student  tracking)  is  not  in  the  table,  it  means  that  it  is  not  statistically 

significant  (p  is  greater  than  5%). 
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Instrument 

Variable  /  Question 

p  /  r 

p  value 

Number  of  different  pages  visited. 

r=  0.702 

<0.001 

Student 

tracking 

Number  or  original  posts  to  the  forums. 

r  =  0.509 

0.004 

p  =  0.560 

0.001 

Student 

tracking 

Number  of  articles  posted  to  the  forums. 

r  =  0.508 

0.004 

p  =  0.591 

0.001 

Student 

tracking 

Number  of  visits  to  the  course  homepage. 

r  =  0.507 

0.004 

p  =  0.398 

0.030 

Student 

tracking 

Total  number  of  hits. 

p  =  0.448 

0.013 

Student 

tracking 

Number  of  the  forum  articles  read. 

p  =  0.441 

0.015 

Module  14 

1 1  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  quiz  feature  being  used  to  teach 
this  course?’ 

p  =  0.465 

0.017 

Student 

tracking 

Number  of  visits  to  die  course  content  page. 

r  =  0.405 

0.025 

■ 

Number  of  follow-up  posts  to  the  forums. 

r  =  0.379 

0.039 

p  =  0.446 

0.013 

Module  14 

13  -  ‘How  would  you  rate  your  comfort  level  at 
this  point  with  word  processing?’ 

p  =  0.395 

0.046 

Table  88.  Significant  Correlations  and  Levels  of  Significance  between  the  Final 
Course  Grades  and  Other  Variables  (Ordered  by  Level  of  Significance). 


Final  Grades  -  Summary 

Final  grades  significantly  correlated  with  ten  variables:  with  eight  online  tracking 
indicators  (i.e.,  all  of  them  used  in  this  research),  and  with  two  measures  of  comfort  level 
(with  the  quiz  feature  and  with  the  word  processing).  The  correlations  do  not  show 
causality,  but  they  describe  relations  between  sets  of  collected  data.  On  the  other  hand, 
this  is  confirmation  that  on-line  tracking  indicators  are  relevant  for  the  research  questions 
in  this  thesis  and  can  be  used  as  important  indicators  of  several  factors  in  a  DL  course  in 
further  studies. 
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E.  INTERVIEWS  WITH  THE  STUDENTS 


Seventeen  out  of  thirty  students  agreed  to  give  interviews. 

The  instructor  thought  that  only  students  with  very  good  course  grades  would 
agree  to  the  interviews.  Statistics  support  that  expectation.  From  seventeen  volunteers, 
nine  had  A  and  eight  A-  as  a  final  grade.  However,  I  wouldn’t  say  that  only  the  best 
students  volunteered,  but  rather  that  the  students  with  some  problems  in  the  course  (those 
with  the  grades  below  A-)  did  not  volunteer  to  participate.  More  than  likely,  this  sample 
of  seventeen  students  is  self-selected. 

Interviews  were  conducted  in  the  quiet  part  of  the  Dudley  Knox  Library  at  NPS. 
Only  the  interviewer  made  notes.  At  the  beginning  of  each  interview,  die  interviewer 
explained  the  purpose  of  the  interviews,  emphasized  that  all  responses  would  be  used 
only  for  the  research,  and  stated  that  the  identity  of  any  student  would  not  be  revealed  in 
the  thesis.  Most  interviews  in  average  lasted  between  fifteen  and  twenty  minutes. 

In  next  eleven  subsections  I  present  student  responses  in  sequence  of  the 
questions,  grouped  by  their  main  meaning. 

1.  First  Question 

The  first  question  was:  ‘In  general,  could  you  estimate  have  you  learned  more 
from  the  on-line  designed  course  than  you  would  learn  from  a  face-to-face  SS3011 
course?  I'm  asking  you  about  the  successful  learning  of  the  subject  matter  taught  through 
distributed  learning,  not  about  a  particular  instructor  or  anything  else.’  There  were  not 
any  questions  similar  to  this  one  in  the  four  applied  on-line  questionnaires  or  in  the  SOFs. 

Here  are  their  full  responses,  grouped  into  three  main  types: 
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No 


•  Clearly  no.  Learned  lot  of  basic  things,  but  question-answer  dialogue  is 
not  good. 

•  Probably  not.  Something  is  missing  -  interaction. 

•  Somewhat  less.  It  was  his21  personal  fault  -  lack  of  motivation  to  learn.  All 
information  was  available,  but  he  was  short  on  time.  It  is  different  when 
one  hears  something  in  a  classroom  -  it  sounds  more  serious. 

•  Definitely  not  more.  It  was  hard  to  read  everything.  Main  text  has  to  be 
read,  and  then  one  usually  waits  for  a  couple  of  days  and  read  the 
comments.  He  had  a  problem  because  he  likes  to  have  'duty  expert' 
available.  The  advantages  of  the  DL  would  be  large  if  he  were  somewhere 
on  duty,  not  in  the  school. 

•  No,  but  he  got  what  he  needed.  It’s  matter  of  student  mind  set;  it’s 
competing.  It’s  overview  of  space  tech;  he  got  what  he  wanted.  If  a  course 
were  more  oriented  to  his  future  job,  he  would  give  more  efforts. 

•  Clearly  no.  He  perceives  face-to-face  instruction  as  more  challenging, 
more  demanding,  and  it  is  giving  more  options  in  what  and  how  to  learn. 

•  Clearly  no.  He  was  exposed  to  more  material,  but  it  was  not  integrated. 

•  Clearly  no.  It  is  not  possible  to  throw  into  DL  course  all  information  from 
a  face-to-face  course  and  expect  it  will  function.  It  all  has  to  be 
reconstructed.  There  was  much  information  in  the  course  (books,  all  those 
modules  and  links).  He  was  missing  prioritization,  i.e.  selection  of 
information. 

•  He  learned  less.  It’s  about  self-discipline.  Students  were  not  forced  to  read 
it  all,  so  it  was  easy  losing  time. 

•  He  would  probably  learn  more  in  face-to-face  class. 

•  Clearly  no.  Students  did  what  they  had  to  do.  She  is  personally  not  very 
interested  for  this  subject  area.  In  a  face-to-face  course,  she  would 
probably  learn  approximately  the  same  or  little  bit  more,  what  depends  on 
instructor. 

Same  or  undecided 

•  Equal,  not  more  or  less.  Organization  was  different,  easier,  but  equal 
results. 

•  It's  hard  to  measure  that.  Equally,  readings  are  giving  lot  of  learning.  It's 
difficult  with  materials  you  dont  understand  and  instructor  has  to  explain  - 
this  was  lacking  in  the  course. 


21 1.e.  it  was  the  fault  of  this  student  who  gave  the  response. 
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Same. 


•  He  learned  approximately  the  same;  either  way  offers  enough 
opportunities.  Maybe  less  interaction  with  the  instructor,  and  overview 
class  was  not  very  deep,  but  it  was  very  good  -  because  this  course  was  an 
introduction  to  the  area. 

More 

•  She  probably  learned  more  from  the  Internet  version.  Content  was  more 
interesting,  broader  pool  of  resources.  Excellent  instructor's  job.  She 
helped  in  learning  process.  It  depends  on  instructor.  Interaction  was  better 
too.  She  personally  likes  to  express  her  opinion  in  calm  environment,  this 
suited  to  her. 

•  Not  in  all  areas,  but  in  some  he  learned  a  lot  more,  especially  about  own 
learning  style  and  similar  personal  learning  characteristics.  He  likes  to 
discuss,  but  he  was  limiting  his  posted  comments  only  to  few  sentences. 
He  learned  a  lot  from  3D  graphics  and  from  links.  He  learned  very  much 
in  visual  sense,  but  not  so  much  in  subject  matter. 


First  Question  -  Summary 

Overall,  their  responses  may  be  summarized  as:  ‘no’  -  eleven  students;  ‘same’  - 
four  students;  ‘yes’  -  two  students.  Qualitatively  analyzing  the  messages  in  the  ‘no’ 
responses,  I  found  three  key  elements  that  students  thought  caused  the  on-line  course  to 
be  less  effective  than  a  face-to-face  course: 

•  The  course  content  and  design  (explanations:  it  wasn’t  integrated; 
dialogue  can’t  replace  full  personal  communication;  missing  selection  of 
information;  DL  course  can’t  work  as  face-to-face  course). 

•  Students  and  their  motivation  (personal  fault;  missing  competition  or 
challenge). 

•  Environment  (it  would  work  better  from  a  duty  than  within  the  school. 


I  found  more  difficult  to  interpret  student  responses  in  the  ‘same’  group.  As  the 
main  justification  for  their  responses,  the  students  reported  that  though  on-line  and  face- 
to-face  courses  have  a  different  organization  of  a  learning  process  (with  different 
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amount  of  interaction),  the  learning  results  (effectiveness)  are  equal.  They  thought  there 
were  enough  learning  opportunities  either  way. 

Two  students  responded  they  learned  more  from  the  on-line  version  of  the  course. 
One  of  them  was  very  positive  about  the  on-line  course,  another  only  partially.  Both 
mentioned  terms  like  Teaming  process’  or  Teaming  style’,  and  emphasized  control  over 
their  interactions  with  the  others,  which  they  considered  as  an  important  and  desirable 
factor  for  their  opinion  about  the  course. 

2.  Second  Question 

The  second  question  addressed  forums:  “Students’  answers  about  forums  were 
very  different,  as  to  whether  forum  exercises  were  useful  in  helping  students  understand 
the  subject  matter.  What  is  your  final  opinion  about  their  real  usefulness  in  facilitating 
your  learning?  Also,  comment  on  how  much  you  liked  using  forums.” 

Here  are  the  student  responses: 


Not  useful 

•  He  didn’t  like  them.  They  were  too  time-consuming:  sit,  read  everything, 
scroll...  Some  were  useful,  some  less  useful,  there  was  too  much  writing. 
It  is  easier  when  the  students  worked  in  small  groups;  otherwise  it's  too 
difficult  to  control  it.  Personally  he  prefers  face-to-face  discussions,  even 
by  VTC  once  a  month. 

•  He  didn’t  like  forums;  didn’t  learn  much  from  them.  Most  of  them  were 
too  focused  on  some  issues,  what  was  not  very  productive.  Instructor 
wasn’t  able  to  moderate  the  forums  well,  to  encourage  and  direct/guide  the 
discussions. 

Probably  useful 

•  Forums  seem  to  be  effective,  useful  interaction  tool,  but  there  is  a 
drawback  -  it’s  more  opinion  oriented  than  factual  learning-based.  People 
have  different  background  and  opinions;  it  was  useful  to  see  that 
difference,  but  he  is  not  sure  how  useful  that  was  for  learning.  If  you  post 
a  comment  without  reading  other  posts,  all  readers  can  see  you  were  not 
prepared. 


215 


Useful 


•  It  would  be  more  useful  in  more  advanced  class.  Good  in  sense  of 
applying  the  learned  about  space  systems,  but  most  of  material  was  brand 
new  to  the  students.  Interesting,  but  people  often  didn’t  know  what  to  say, 
tended  to  go  off  subject. 

•  She  liked  them.  Useful,  good  references,  it's  possible  to  go  back  to  them. 
People  normally  oppose  the  change  in  way  they  work/function  as  students. 
The  course  was  well  organized;  contained  lot  of  thoughts. 

•  It  was  useful  for  learning.  He  liked  forums.  Timing  was  wrong  in  this  type 
of  audience,  in  context  of  NPS:  it  should  be  Monday-Friday. 

•  It  was  more  useful  in  small  groups;  he  liked  that.  However,  the  instructor 
was  the  expert,  not  the  students,  so  it  seemed  that  real  knowledge  wasn’t 
‘part  of  the  game’.  It  was  based  more  as  ‘opinion  section’,  not  factually 
based  materials. 

•  With  the  given  limitations  of  on-line  systems,  forums  were  useful.  People 
have  different  backgrounds  and  opinions,  what  he22  likes  and  thinks  that 
was  very  useful.  There  were  some  problems  here  and  there;  posts  should 
be  limited  by  length. 

•  Yes,  useful,  he  got  majority  of  what  he  gained  it  the  course  from  reading 
forum  messages,  even  though  it  was  not  much  a  direct  learning.  He 
enjoyed  them,  liked  them.  He  learned  mostly  from  texts. 

•  Beneficial,  even  more  after  dividing  the  class  into  smaller  groups.  It  takes 
time,  but  this  is  closest  to  classroom  interaction  you  can  get.  He  liked 
forums. 

•  Useful.  He  recommended  division  of  forums  into  smaller  groups,  what  the 
instructor  did  later.  Many  students  posted  because  of  grades;  they  knew  it 
was  expected  of  them.  Some  had  excellent  ideas,  but  they  wrote  too  much. 
Instructor  should  be  more  specific  about  topics  of  the  forums  and  should 
limit  the  length  of  posts.  Many  thoughts  are  being  repeated.  Maybe  forums 
should  be  made  only  for  some  of  the  modules. 

•  He  liked  them  -  he  heard  different  opinions,  everybody  thought 
differently.  When  they  were  easy  for  use  and  functional  -  it  was  very 
good.  When  there  were  technical  problems  with  posting  to  forums,  there 
was  no  use  of  them. 

•  Yes,  they  were  useful,  one  of  the  highlights  of  the  course.  He  liked  them. 
Some  people  were  very  active  in  forums,  have  written  a  lot. 


22  He  -  the  interviewed  student. 
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•  She  liked  them.  They  were  useful.  They  forced  students  not  to  just  react, 
but  to  form  an  opinion  and  to  ask  for  more  explanations.  Some  people  are 
not  comfortable  speaking  in  class,  but  in  forums  they  gave  more,  they 
participated  more.  Forums  helped  to  get  together  more  different  opinions 
and  to  explore  different  opinions.  It  was  different  than  class  environment. 

•  Yes,  useful,  but  he  didn't  like  them.  Some  students  had  excess  time  -  they 
wrote  very  much.  He  wrote  brief  texts.  He  didn’t  like  lack  of 
responsiveness  in  the  forums. 

Very  useful 

•  She  liked  them.  Open  dialogue,  and  people  more  forthcoming  with  their 
opinions.  She  would  prefer  chat  (real  time)  as  addition.  Very  useful, 
especially  towards  the  end.  All  in  all,  very  well  rounded  opinion. 

•  He  saw  forums  as  very  useful;  lot  of  reading  involved,  challenging 
because  one  has  to  be  careful  what  to  write.  Weakness  is  that  you  should 
read  all  previous  postings.  There  was  no  way  to  search  all  posted 
information;  it  would  be  a  very  useful  capability,  way  to  cataloguing 
posts.  Personally,  he  works  a  lot  on  development  of  web  applications. 


Second  Question  -  Summary 

If  we  reduce  the  student  responses  to  this  question  in  the  interviews  to  key 
messages/responses,  they  would  be:  ‘not  useful’  -  two  responses;  ‘probably  useful’  -  one 
response;  useful  —  12  responses;  and  ‘very  useful’  -  two  responses.  Analysis  of  the 
responses  suggests  that  the  students  used  the  same  argument  about  student  opinions  to 
support  quite  opposite  responses.  Here  are  the  factors  they  emphasized  for  their  responses 
to  this  question: 


Responses  ‘not  useful’ 

•  Not  much  to  learn  because  posts  are  opinion-driven,  too  focused,  not 
factual’. 

•  Too  time  consuming. 

Responses  ‘useful’ 

•  Diverse,  opinion-driven  and  background-diverse  posts.  (Five  responses.) 
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•  Post  can  serve  as  good  references. 

•  Forums  acted  as  replacement  for  class  interaction. 

•  Forums  help  some  students,  otherwise  very  quiet,  to  express  more  freely. 

•  Posts  more  discuss  applications  than  theoretical  aspects  of  the  space 
systems  issues. 

Response  ‘probably  useful’ 

•  Dilemma  (or  drawback):  forums  may  be  useful  or  not  because  they  are 

opinion-driven. 

Responses  ‘very  useful’ 

•  Students  are  forthcoming  with  their  opinions  -  open  dialogue. 


3.  Third  Question 

The  third  question  addressed  quizzes:  ‘Same  question  about  Quizzes  -  please 
comment  on  how  useful  they  were  for  you  and  how  much  you  liked  them!’ 


Here  are  the  responses  to  the  third  question: 

Not  useful 

•  Not  useful.  They  didn’t  have  enough  depth.  It  was  possible  to  answer  by 
reading  from  the  textbook.  He  didn’t  like  them. 

•  Absolutely  not  useful.  Too  easy,  they23  could  do  five  runs,  with  the 
book.24  They  did  not  demand  students  to  think.  She  didn’t  mind  doing 
them.  Maybe  some  essay-type  questions  could  be  added,  especially  in 
second  half  of  the  course. 

Not  very  useful 

•  Not  very  useful.  They  were  never  completely  fixed.  They  were  easy,  but 
one  had  to  read  the  book.  They  should  be  more  challenging. 


23  The  students. 

24  Quizzes  provided  students  with  feedback  about  incorrectly  answered  questions,  and  enabled  them  to 
go  through  all  questions  up  to  five  times.  Because  quizzes  were  on-line,  it  was  possible  for  students  to  use 
the  textbook  and  notes  while  answering  quiz  questions. 
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•  Not  very  useful.  It  was  more  like  a  test  have  you  read  the  material  at  all, 
than  anything  else.  It  would  be  better  if  quizzes  were  directed  towards 
some  knowledge  test  at  the  end  of  the  course. 

•  If  open  book  and  more  than  one  pass,  not  much  use. 

•  They  seemed  pointless;  they  didn’t  capture  students’  understanding  of 
subject. 

•  It  was  just  a  drill,  not  very  useful:  open  book,  no  restrictions,  almost 
anyone  with  the  book  could  do  to  them. 

•  He  liked  them  in  sense  of  not  demanding  too  much,  but  the  facts  from 
quizzes  didn’t  remain  in  knowledge;  in  essence  it  had  not  much  use. 

Useful 

•  Good  as  a  review  process.  Open-book  so  student  could  choose  not  to 
really  learn  about  topics  from  quizzes  but  simply  to  read  from  the  book. 
Problems  with  grading  were  very  frustrating.  It  was  too  easy,  but  good 
subject  matter  issues  were  present  in  content  of  the  quizzes.  The  quizzes 
were  too  simple. 

•  Yes,  useful,  forcing  you  to  read.  That  was  the  point.  Easy. 

•  They  were  not  difficult;  he  liked  them  because  they  contained  information 
and  facts,  not  opinions. 

•  Yes,  useful,  forcing  students  to  do  the  reading.  For  DL  (web),  self- 
discipline  is  very  important.  Quizzes  were  pretty  relevant.  Otherwise  it 
would  be  difficult  for  a  student  to  identify  where  he  stands,  what  he  has  to 
do.  They  were  definitely  a  roadmap  of  the  course,  important  feedback. 

•  Yes,  they  were  useful,  but  not  fair  in  sense  that  they  can  be  answered  in 
collaboration  with  other  students.  They  should  be  more  challenging,  but 
they  were  not  too  easy.  He  neither  likes  nor  dislikes  quizzes. 

•  Useful,  they  let  students  know  what  they  should  have  learned.  Make 
students  to  HAVE  to  read/leam  material.  He  didn't  like  them  much;  he's 
neutral.  To  have  several  runs  in  a  quiz  is  counterproductive.  Quizzes  didn't 
demand  true  understanding  from  the  students. 

•  Quizzes  were  appropriate.  They  help  to  focus  study,  help  to  understand 
what's  important,  what  you  needed  to  learn.  They  do  not  need  to  be 
difficult  to  do  that.  He  liked  them. 

•  They  were  too  easy.  There  were  some  technical  problems  at  the  beginning. 
It  is  good  that  more  runs  are  possible  in  quizzes,  but  then  their  usefulness 
(and  fairness)  becomes  questionable.  The  quizzes  should  be  a  little  deeper, 
more  comprehensive.  Maybe  only  two  tries,  not  five.  Also,  maybe  few 
essay-type  questions  should  be  added. 
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•  They  were  tied  closely  to  the  book,  without  them  students  wouldn’t  read 
as  much.  Quizzes  were  good,  forces  you  to  learn  but  not  too  hard.  Helpful 
-  forced  students  to  self-organize. 

•  He  liked  quizzes  and  there  should  be  more  of  them;  otherwise  he  would 
study  too  much  in  last  minute.  He  liked  the  possibility  of  redoing  the  quiz. 
They  are  useful  for  the  learning  process.  Bugs  made  some  of  the  students 
angry.  Quizzes  were  ‘confidence  builders’;  they  were  balanced  mixture  of 
course  content,  some  maybe  even  too  condensed. 

Third  Question  -  Summary 

Summarized,  student  responses  were:  ‘not  useful’  -  two  responses;  ‘not  very 
useful’  -  four  responses;  unclear  -  one  response;  ‘useful’  -  ten  responses.  After  analyzing 
these  responses,  I  concluded  that  the  key  reason  for  such  opposite  views  about  the 
quizzes  was  different  interpretation  of  the  role  of  the  quizzes.  For  instance,  both 
groups  of  students  who  saw  the  quizzes  as  either  not  useful  or  as  useful  mentioned  that 
they  were  easy.  However,  they  saw  role  of  quizzes  in  the  course  completely  differently. 
Also,  some  students  thought  the  quizzes  were  easy  but  did  not  like  them.  These 
students  were  used  to  competitive,  demanding,  and  challenging  tests,  quizzes,  and  exams. 
Consequently,  these  quizzes  ("with  five  possible  tries  and  with  no  time  or  closed-book 
constraints)  were  not  challenging  them.  In  fact,  quizzes  that  are  too  easy  may  seem 
offensive  to  them. 

4.  Fourth  Question 

The  fourth  through  seventh  question  covered  issues  of  interactivity.  The 
statements  asked  the  students  to  comment  the  amount  or  quality  of  the  interaction 
between  the  students  and  the  instructor  or  between  the  students. 

The  fourth  question  was:  “Students’  answers  to  the  question  about  how  they  saw 
the  amount  of  interaction  with  the  instructor  dining  the  SS301 1  course,  compared  to  a 
similar  resident  (face-to-face)  course,  were  very  different.  Please  provide  your  view.” 
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are  the  responses  to  fourth  question: 


Much  less 

Much  less  interaction.  She  felt  depersonalization.  The  instructor  didn't 
know  some  of  the  students.  Not  much  one-to-one  interaction. 

A  lot  was  lost.  No  body  language.  It  was  time  consuming,  but  the  students 
lost  important  part  of  communication. 

Much  less  interaction  in  on-line  than  students  needed.  The  instructor 
worked  hard  ‘behind  the  curtain’.  At  the  beginning  the  instructor  posted 
many  comments,  but  later  less  -  it  was  lacking. 

Much  less.  (Two  responses). 

Much  less  amount.  Personally  he  likes  to  talk  a  lot,  to  personally  ask  the 
instructor.  His  interaction  was  clearly  less. 


Less 

Not  enough,  maybe  the  instructor  wanted  to  be  ‘invisible’  too  much. 

It  was  less,  but  it  wasn’t  so  bad.  It’s  simply  different  environment  (for  the 
students). 

Significantly  less. 

Less.  There  is  no  personal  interaction  like  in  a  classroom,  but  the 
instructor  can  better  focus  the  interaction  he/she  has,  and  to  use  it  more 
efficiently. 

The  problem  was  that  DL  course  was  organized  in  local  environment.  If 
students  had  to  take  written  materials  from  the  instructor,  then  DL  looses 
its  sense.  The  instructor  was  giving  her  best,  responded  to  all  e-mail 
messages,  talked  a  lot,  and  that  was  very  good.  The  amount  (of  interaction 
of  students  with  the  instructor)  was  a  little  less. 

Less. 

Less.  Most  of  interaction  in  forums  was  about  technical  problems  The 
instructor  pretty  often  participated  in  the  forums,  mostly  in  form  of 
response  to  one  person's  post. 

Not  enough. 


Unclear 


•  It  is  an  individual  interaction  with  the  instructor.  In  face-to-face  courses, 
students  rarely  exchange  e-mail  with  instructors.  That  was  the  good  side  of 
on-line  version  of  this  course. 


Enough 

•  There  was  enough  interaction;  it  was  possible  to  contact  the  instructor 
anytime,  or  other  students. 

More 

•  In  fact,  there  was  MORE  interaction.  He  had  another  on-line  course,  very 
structured,  without  almost  any  interaction  with  the  instructor.  In  SS3011, 
the  instructor  was  very  dynamic,  stimulative,  responsive,  and  she  was  key 
factor  for  the  success  of  the  course. 


Fourth  Question  -  Summary 

Reponses  may  be  summarized  as  following:  ‘much  less’  -  six  comments;  ‘less’  - 
eight  comments;  unclear  -  one  comment;  ‘enough’  -  one  comment;  ‘more’  -  one 
comment.  Analyzing  the  ‘much  less’  responses,  I  concluded  that  those  students  felt 
uncomfortable  because  of  the  absence  of  the  instructor’s  personal  presence.  They 
mentioned  terms  like  ‘body  language’,  ‘depersonalization’,  and  similar  terms  without  any 
other  clear  justifications  for  such  responses.  The  student  who  responded  ‘more’  in  fact 
was  comparing  this  course  to  another  on-line  course,  but  with  much  less  interactivity. 
Other  students  gave  a  wide  spectrum  of  explanations.  It  seems  that  their  approach  to  DL 
interactivity  was  more  diverse  than  approach  of  the  students  who  responded  ‘much  less’. 
5.  Fifth  Question 

The  next  statement  was:  ‘I  want  to  ask  you  similar  question  about  the  quality  of 
students’  interaction  with  the  instructor,  compared  to  a  resident  course.  How  did  you 
see  it?’ 
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Here  are  full  responses  of  the  students  to  fifth  question: 

Less 

•  It  decreases  if  it’s  only  all  in  words.  With  multimedia  and  similar 
improvements,  it’s  better,  but  there  are  more  lost  aspects.  Very  time 
consuming.  It’s  much  better  in  small  groups,  more  personable. 

•  Usually  someone’s  opinion  can  be  understood  better  if  you’re  facing  that 
person.  Otherwise,  that  is  lost.  The  instructor  wasn’t  present  enough. 
She25  ‘intervened  more  after  the  fact’  than  she  was  engaged. 

•  It  depends  on  instructor.  If  the  instructor  is  active,  reactive,  responsive, 
then  it's  OK.  Maybe  little  bit  lower  quality  than  in  face-to-face  course. 

•  Clearly  less  effective.  Instructor  was  relatively  OK,  trying  to  be  involved 
as  much  as  possible.  IW26  class  -  the  students  barely  heard  from  the 
instructor  at  all,  that  was  hands-off  process. 

•  Nothing  can  really  replace  face-to-face,  even  though  the  instructor  was 
very  responsive.  He  felt  the  need  to  talk  to  her. 

Unclear 

•  The  instructor  was  very  good;  she  was  able  to  answer  every  question,  what 
increases  quality.  However,  she  didn’t  force  all  students  to  enter  into 
interaction  with  her.  Deeper  integration  is  missing,  nonverbal 
communication  not  present. 

•  That’s  tough  to  achieve.  The  instructor  was  very  responsive,  but  maybe 
not  enough  for  him  personally.  He  felt  something  was  missing  to  conclude 
each  of  the  modules,  maybe  real-time  on-line  (chat)  session  about  it,  or  at 
least  the  instructor  sending  some  kind  of  conclusion  to  each  student  (some 
‘walk-away’  points). 


Same 

Probably  the  same  as  in  face-to-face.  The  students  had  possibility  to  visit 
personally  or  contact  by  phone  the  instructor,  so  they  were  able  to  increase 
quantity  and  quality  of  this  interaction  if  they  needed  to. 

Good 


25  The  instructor. 

26  Information  warfare. 
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•  The  class  was  too  large.  Instructor  didn’t  have  enough  time.  Quality  was 
good. 

•  Good  quality.  The  instructor’s  comments  were  good,  relevant. 

•  It  was  better  than  in  face-to-face;  it  was  individual  feedback. 

•  Interaction  with  the  instructor  was  mostly  by  e-mail,  but  it  was  good. 
Whenever  he  personally  met  instructor,  interaction  immediately  started. 
There  were  no  occasions  when  the  instructor  didn't  respond. 

•  The  quality  was  pretty  good.  However,  responses  from  the  instructor 
should  be  quicker. 

•  It  was  fine;  the  instructor  was  sending  comments  throughout,  but  it’s 
limited. 

•  It  was  good.  The  instructor  was  responsive,  though  not  quite  enough.  For 
him  personally,  face-to-face  is  ‘better  quality’  because  of  his  learning 
style.  In  face-to-face  class,  stupid  question  is  quickly  forgotten,  but  in  DL 
it  stays  on  server.  In  terms  of  contacting  with  people  from  space  systems 
practice,  nothing  is  good  as  DL. 

Better 

•  The  quality  is  better.  There  is  always  an  option  to  initiate  more  interaction 
if  you  need  it.  There  is  a  potential  here  for  much  more  interaction  (like  in 
face-to-face),  but  it's  not  really  as  necessary. 

•  Not  diminished,  maybe  even  better.  She27  had  problems  with  the  password 
(SSN).  The  instructor  was  doing  her  best.  She  was  very  responsive. 


Fifth  Question  -  Summary 

Summarized,  the  responses  were:  ‘less’  —  five  responses;  unclear  —  three 
responses;  ‘good’  -  seven  responses;  ‘better’  -  two  responses.  Explaining  the  ‘less’ 
responses  (there  were  five  such  opinions),  students  emphasized  the  lack  of  the 
instructor’s  personal  presence,  which  was  also  similar  to  the  explanations  for  the  ‘much 
less’  responses.  Ten  students  thought  that  the  quality  of  the  interaction  was  good,  the 
same,  or  even  better  than  in  the  face-to-face  class.  Even  though  many  of  these  ten 
students  mentioned  other  problems  such  as  too  large  a  class,  problems  with  passwords, 


27  The  student. 
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the  instructor’s  responses  was  not  quick  enough,  personal  preference  with  face-to-face 
classes,  they  still  saw  the  quality  of  the  interaction  better  anyway. 

6.  Sixth  Question 

The  sixth  question  asked:  ‘One  of  most  controversial  questions,  according  to  the 
students’  answers,  was  about  the  amount  of  interaction  between  the  students, 
compared  to  a  face-to-face  course.  Some  thought  SS3011  offered  significantly  less 
interaction,  some  others  -  significantly  more.  What’s  your  comment?’ 


The  complete  responses  to  sixth  question  were: 

Much  less 

•  Much  less  interaction  than  in  face-to-face  courses.  Majority  of  interaction 
was  between  the  students.  Most  of  interaction  about  the  course  in  the  on¬ 
line  mode  was  about  the  course  itself,  not  about  topics. 

•  Much  less  -  he  met  almost  nobody,  only  one 

•  Very  low  amount 

Less 

•  For  majority  of  students  it  was  purely  on-line.  Amount  is  limited. 

•  Significantly  less  than  in  face-to-face  course.  He  was  thinking  a  lot  about 
applicability  of  this  knowledge,  he  was  expecting  a  ’toolbox’  from  the 
course.  Students  talked  about  that  a  lot. 

•  Less  than  in  normal  class.  Tempo  was  fragmented;  many  were  working 
only  once  a  week.  There  were  many  postings;  it  was  expected  from  the 
students.  The  quantity  was  not  at  the  same  level  as  quality. 

•  Forums  were  there  for  this  interaction.  Of  course  it  was  different  than  in  a 
classroom  (face-to-face).  Less  interaction. 

•  Besides  forums,  not  much  interaction.  DL  form  decreased  interaction. 

Little  less 

•  Probably  little  less  because  there  was  less  informal  interaction  which 
usually  goes  in  a  classroom.  He  didn’t  have  any  informal  communication. 
Somewhat  less  than  in  face-to-face  course. 
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•  Not  so  bad.  He  felt  good  and  bad  sides  that;  maybe  bad  sides  a  little  bit 
more.  If  he  were  on  duty,  on  a  ship  or  something  like  that,  it  would  be 
OK!  Here  in  school  occasionally  it  was  frustrating.  It  even  happened  that 
students  were  sitting  one  next  to  the  other  and  they  were  exchanging  e- 
mail! 

•  Maybe  the  same  or  little  bit  less.  If  students  sit  together  in  a  classroom,  of 
course  they  exchange  more  information. 

Same  and  unclear 

•  Purely  within  the  forums  (at  least  for  her).  Pretty  good  interaction. 
Everybody  can  respond  to  everybody.  On-line  delivery  increased  quality 
and  quantity  of  interaction  between  the  students. 

•  About  the  same.  Students  in  NPS  talked  at  other  classes  too.  Mostly,  it 
was  OK. 

More 

•  The  difference  in  interaction  was  almost  unfair.  There  was  a  lot  of 
interaction,  because  the  students  were  forced  to  respond  to  posts. 

•  There  was  more  interaction  here;  in  face-to-face  courses  every  student  is 
concentrated  on  the  instructor.  SS3011  students  were  with  different 
backgrounds,  but  exposed  to  the  same  material  and  they  became  closer. 
Maybe  the  instructor  didn’t  mediate  enough;  she  ought  to  direct  more 
attention  towards  the  key  points. 

•  More,  because  it  was  required  to  post  responses  and  thoughts.  In  usual 
classroom  there  are  always  some  students  who  do  not  talk  much.  Here  all 
students  were  forced  to  participate,  so  there  is  more  interaction  than  in 
face-to-face  class.  Good  quality  and  quantity  of  interaction  between  the 
students,  but  for  a  high  price  (again,  he  personally  prefers  ‘face-to-face’). 
He  simply  doesn’t  like  to  write. 


Sixth  Question  -  Summary 

I  grouped  student  responses  by  their  key  messages  as  follows:  ‘much  less’  -  three 

responses;  ‘less’  -  five  responses;  ‘little  less’  -  three  responses;  ‘same’  or  unclear  -  two 

responses;  ‘more’  -  three  responses.  Interestingly,  there  was  a  broad  spectrum  of 

different  answers  to  this  question.  Most  students  (eleven  out  of  seventeen),  thought  that 

there  was  interaction  between  the  DL  students  compared  to  a  face-to-face  course.  It  is 
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very  difficult  to  interpret  this  data,  but  it  seems  that  these  student  subgroups  with 
different  opinion  might  have  a  different  focus  in  face-to-face  classes:  one  focused  their 
attention  more  on  the  course  instructor,  and  the  other  more  on  the  teaching  material  in  the 
course.  If  so,  their  perception  of  the  change  of  interactivity  in  the  DL  environment  might 
be  so  opposite,  as  it  was  in  the  example  of  the  responses  to  this  statement. 

7.  Seventh  Question 

The  next  question  asked:  ‘The  last  question  about  interaction  covers  the  issue  of 
quality  of  interaction  between  the  students,  what  is  important  for  the  learning  process.28 
How  would  you  comment  it,  in  sense  of  comparing  the  quality  of  interaction  between 
students  in  on-line  SS3011  with  interaction  between  students  in  some  similar  face-to- 
face  course?’ 


Here  is  how  the  students  responded  to  seventh  question: 

Worse 

•  Worse,  same  answer  as  in  question  6.  Synergy  never  develops. 

•  The  quality  was  suffering,  but  it  wasn't  because  of  the  instructor  but 
because  of  media  used  (DL). 

•  The  interaction  quality  was  not  as  good  as  in  a  face-to-face  course. 
Limited,  time-constrained.  No  real-time  interaction.  The  best  sort  of 
learning  interactivity  is  arguing  in  class. 

Same  or  uuclear 

•  Fair,  in  the  middle  of  the  scale  (2.5/5).  Before  the  end  of  the  course  he 
understood  the  instructor's  expectations/demands,  but  that  can't  be  done 
mechanically,  a  dialogue  is  necessary.  When  one  starts  to  feel  that  he  isn’t 
getting  anything  more  from  new  postings,  then  there  is  no  more  need  to  be 
engaged  in  that  dialogue.  At  one  moment,  it  isn’t  stimulative  any  more. 
Time  factor  is  very  important 


28  Research  and  experiences  with  DL  at  UCF  and  the  instructor’s  and  student  statements  during  this 
course  agreed  about  the  importance  of  interactivity  for  efficient  learning  in  a  DL  course. 
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It  depends:  some  students  thought  well  before  posting  an  answer.  Others 
went  directly  with  their  opinion,  without  many  facts  in  the  content. 

Forums  were  best  part  of  interactivity  in  this  course.  They  reflect  that 
quality  of  interaction.  Some  students  were  expressing  own  opinions; 
others  were  reacting,  or  were  not  able  to  react  if  everything  is  already  said. 

Some  people  have  more  difficulty  talking  in  front  of  the  others.  Also,  if 
on-line,  one  has  to  be  careful  what  he  is  writing.  It  would  be  better  if  an 
instructor  told  to  students  to  feel  free  to  write  their  own  opinion  (with  a 
level  of  privacy  too!),  and  it  would  not  be  sent  to  anybody  else,  but  she 
can’t  remember  that  the  instructor  in  this  course  told  anything  like  that. 

It  was  laborious.  He  would  like  more  real-time  discussion.  There  was 
quality,  but  maybe  not  in  right  format. 

Little  better 

Probably  little  higher  quality,  because  one  has  to  think  well  before  posting 
message/  comment. 


Better 

Pretty  good  quality.  Casual  interaction  from  face-to-face  courses  was 
lacking  here. 

Pretty  good 

Quality  was  better.  One  has  to  research  what  he  wants  to  say  -  can’t  write 
too  much.  He  personally  used  to  revise  his  posts  before  sending. 

Academically,  the  quality  was  better.  It  was  also  good  that  posts  remain  as 
references.  So  one  has  to  be  careful  what  to  post,  have  to  check  it  in  the 
book. 

It  was  hard,  but  probably  better  because  of  much  written  communication. 

Yes,  better  quality,  but  different  kind  of  interaction.  Personally  he  prefers 
not  to  write,  so  he  was  very  careful  in  writing  -  that  was  a  challenge  for 
him. 

Probably  higher  quality.  The  instructor  never  directly  commented  on 
postings. 

He  agrees  that  one  should  think  before  posting  (it  stays  on  server),  but 
there  are  some  students  who  posted  (in  forums)  only  because  they  had  to. 
Quality  was  higher. 


Seventh  Question  -  Summary 

Summarized,  the  student  responses  were:  ‘worse’  -  three  responses;  ‘same’  or 
unclear  -  five  responses;  ‘little  better’  -  one  response;  ‘better’  -  eight  responses.  Clearly, 
when  the  question  is  about  the  quality  of  the  interaction  between  the  students  in  the  DL 
environment,  the  students  have  significantly  more  positive  than  negative  opinions, 
compared  to  the  question  on  the  amount  of  the  same  type  of  interaction.  Nine  students 
thought  the  quality  of  the  interaction  was  better  in  DL  than  in  a  face-to-face  course,  and 
only  three  students  thought  the  opposite.  Several  students  stated  that  one  of  the  key 
reasons  for  this  opinion  was  the  fact  that  posted  forum  articles  are  written  material  that 
stays  on  the  server  as  a  reference.  Therefore,  it  is  necessary  to  be  careful  about  what  is 
posted. 


8.  Eighth  Question 

The  eighth  question  in  the  interviews  was:  ‘If  you  have  an  opportunity  to  choose 
between  on-line  and  face-to-face  section  of  some  other  class  in  your  curriculum,  what 
would  you  do?  (Examples:  I  would  surely  go  online;  depends  on  the  nature  of  the  course; 
depends  oh  my  schedule;  surely  stay  in  face-to-face  section...)’ 

Their  responses  to  eighth  question,  grouped,  were: 

Face-to-face 

•  While  in  NPS,  preferably  face-to-face,  but  not  100%.  In  some  situation 
DL  gives  far  better  flexibility,  but  the  quality  is  important  issue  too.  Face- 
to-face  has  some  big  advantages.  If  he  is  in  some  degree  education 
program,  it  depends;  he  would  probably  take  it  100%  on-line. 

•  Face  to  face,  mostly.  He  has  a  face-to-face  course  that  is  perfect  for  on¬ 
line  delivering. 

•  Clearly  face-to-face,  it  fits  better  his  learning  style.  He  gains  a  lot  from 
interactions  than  from  quantity. 

•  In  the  school  he  would  take  face-to-face,  but  if  out  of  the  school,  or  taking 
only  some  single  external  course,  he’d  take  on-line. 
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In  NPS  -  face-to-face.  If  on  duty,  of  course  or  traveling  who  knows  where 
-  he  would  take  on-line. 


Depends 

It  depends  on  type  of  a  course.  History  or  similar  (one-way  dialogue)  -  on¬ 
line  is  great.  Math  or  similar  courses  demand  interaction,  so  he  would 
prefer  face-to-face. 

Depends  on  the  course.  More  complex,  dynamic  courses  would  be  very 
hard  on-line.  More  basic,  core  courses  can  be  on-line.  She  had  Information 
Warfare  course  on-line;  they  were  conceptualized  differently. 

It  depends  on  the  course.  If  something  from  history  or  political  sciences, 
surely  on-line.  Math:  face-to-face.  SS301 1  is  somewhere  in  the  middle.  It 
suffered  a  little  as  on-line,  and  it  depends  much  on  depth  of  solving 
technical  problems  and  similar  issues.  The  course  was  more  at  a  general 
level. 

Depends  on  the  course.  Math  -  face-to-face.  This  course  -  OK  for  on-line. 

It  depends  on:  1.  Course.  2.  Instructor  (what  he  knows  about  instructor).  3. 
Time  available.  Example:  now  he  has  a  course,  which  is  all  in  Power  Point 
format.  Such  course  should  be  on-line,  because  course  instructor  is  not 
necessary.  SS301 1 :  students  and  the  instructor  should  meet  at  least  once  a 
week,  even  using  VTC. 

Depends  on  the  content.  Some  courses  like  history  or  social  sciences  are 
better  on-line.  Math  and  similar  -  better  face-to-face. 

If  course  is  similar  to  this  one,  he  would  take  it  on-line,  because  it  gives  a 
lot  of  flexibility.  This  is  the  main  advantage.  If  something  is  more 
difficult,  like  math,  he’d  take  it  face-to-face. 

It  depends  on  course  material.  More  technical  courses  (math,  engineering) 
-  he  would  choose  traditional  classroom,  because  he  is  not  as  experienced 
in  that  area.  He  needs  advantages  of  a  face-to-face  class. 

Depends  on  type  of  a  course  and  on  the  environment  (in  the  school  or  off 
school).  In  the  school  he  would  mostly  choose  a  face-to-face  section.  If  on 
job,  he  would  clearly  choose  on-line,  because  of  flexibility. 

Depends  on  instructor.  Short  courses  (compressed  into  three  days  or  so) 
should  be  taught  via  DL,  but  if  three  weeks  or  more  -  face-to-face.  After 
some  time  interest  decreases.  If  a  course  is  not  demanding,  attention  and 
interest  diminish. 

Probably  face-to-face,  but  it  depends  on  the  course.  If  something  is  not 
interesting  to  him,  DL  wouldn’t  help. 
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•  It  depends  on  the  nature  of  a  course.  If  it’s  required  for  a  degree,  but  she  is 
not  particularly  interested  in  that  course,  she  would  take  it  on-line.  If  it’s 
one  of  more  interrelated  courses  (math  or  similar),  she  would  take  it  face- 
to-face,  especially  if  she  doesn’t  have  background  in  that  area.  In  any  case, 
it’s  GOOD  option! 

Eighth  Question  -  Summary 

Of  the  seventeen  students  interviewed,  five  responded  ‘face-to-face’  and  twelve 
responded  ‘depends’.  Only  five  students  said  they  preferred  the  ‘face-to-face’  version  of 
the  courses,  but  even  within  that  group,  four  students  added  that  they  would  change  their 
opinion  if  they  were  not  at  NPS. 


The  students  who  responded  ‘depends’  gave  three  typical  explanations  of  what 
‘depends’  meant  to  them:  on  the  environment  (in  a  school  or  from  duty),  on  the  nature  of 
a  course  (if  a  less  mathematical  or  technical  course  -  the  student  would  more  likely  take 
the  on-line  version),  and  on  an  instructor. 

9.  Ninth  Question 

The  ninth  interview  item  asked  students:  ‘Please  tell  us  any 
advice/recommendation/suggestion  you  have  for  CDR  Higgins  or  any  other  NPS  or 
Navy  distributed  learning  instructor.’ 

•  Have  to  have  face-to-face  meetings  every  14  days  or  so,  if  possible.  Tech 
things  should  be  fixed  before  the  start  of  the  course  -  students  felt  like 
subject  of  experiment. 

•  The  instructor  was  very  responsive,  and  that  was  good.  Instructor  has  to  be 
very  involved,  has  to  guide. 

•  He  liked  forums.  However,  a  student  has  to  be  careful  with  time  dedicated 
to  the  forums  because  of  other  courses.  Quizzes  shouldn’t  be  too  easy  it 
they’re  on-line.  Face-to-face  quizzes  are  twice  as  difficult.  There  were 
some  tech  problems  at  the  beginning.  The  instructor’s  feedback  has  to  be 
sent  to  individual  students  or  groups.  CDR  Higging  made  some 
observations  about  whole  class,  but  nothing  more  than  that. 

•  There  must  a  backup  plan,  even  a  second  backup  for  the  course  too. 
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•  Advantage  of  DL  is  time  (whenever  you  choose).  So,  entire  course  has  to 
be  ready  and  available  in  advance  (or  least,  large  parts  of  the  course). 

•  On-line  courses  have  to  be  non  math-intensive  or  similar  types  of  classes 
(with  good  textbook). 

•  It  is  the  key  to  combine  DL  and  face-to-face  component  in  an  ideal  ratio. 
That  ratio  depends  on  a  particular  course  in  question.  Also,  when  putting 
links  on  a  page,  instructor  has  to  be  aware  of  total  amount  of  information. 

•  Closeout  of  the  modules  is  important.  He  is  visual  type,  so  he  would  like 
to  occasionally  see  somebody  talk  about  the  issues  in  the  course,  even  via 
video-stream  or  something  like  that. 

•  On-line  learning  is  good;  it  depends  on  subject  matter.  This  course  was 
really  good.  If  there  were  more  math,  it  would  be  too  difficult.  It  depends 
on  a  course.  It  worked  pretty  well. 

•  It’s  important  to  capture  well  and  to  transform  the  knowledge  into  on-line 
content,  but  it’s  important  to  be  able  to  transfer  that  to  both  instructor  and 
the  students.  On-line  learning  misses  sharing  of  who  knows  what  on 
important  issues.  Information  has  to  be  customized.  SS301 1  was  better  in 
on-line  form  probably  because  the  students  were  forced  to  read  all 
material.  Technology  had  to  be  prepared  better. 

•  The  instructor  tried  to  get  feedback  from  the  students  and  was  very  opened 
to  suggestions.  The  feedback  is  very  useful  for  fixing  the  course  problems. 
Right  now  there  is  no  perfect  course  on-line.  There  should  be  more 
multimedia  (applications,  demos,  interactive  things),  and  some  real-time 
chat,  maybe  even  in  voice  transfer  (some  people  are  well  spoken  but  can’t 
write  very  well). 

•  Instructor  should  encourage  students  not  to  wait  to  the  last  moment  (with 
studying).  This  course’s  web  pages  were  predominantly  text-oriented.  He 
had  another  course  with  much  more  graphical  materials  on  web  pages.  He 
thinks  that  more  interactive  and  graphical  content  should  be  put  on 
SS3011  pages  too.  He  is  indifferent  about  real-time  chat  option,  that 
would  be  like  imitation  of  real  classroom  and  DL  wouldn’t  gain  much 
with  it. 

•  Instructor  should  meet  with  students  in  a  classroom  from  time  to  time,  at 
least  couple  of  times,  maybe  in  a  round-table  forum  or  in  small  groups. 
Such  modification  would  facilitate  learning  process.  Another  problem  is 
that  students  do  not  see  such  course  as  high  priority.  List  of  all 
deliverables  should  be  defined  at  the  beginning  of  the  course.  That  should 
be  very  useful. 

•  Do  not  teach  DL  the  same  way  as  face-to-face  learning.  Information  has  to 
be  well  structured. 
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•  Maybe  forums  should  be  real-time  or  both.  Maybe  groups  should  be 
changing,  and  they  should  meet  in  real  time.  Personal  self-dedication  is  a 
necessity. 

•  The  course  was  OK.  It  would  probably  be  more  useful  to  a  people  outside 
of  NPS  -  and  it  should  involve  more  people.  On-line  section  has  to  have 
meetings  too,  maybe  2-3,  especially  wrap-up  discussion. 

•  The  only  complaint  she  has  (what  also  holds  for  some  face-to-face 
courses)  is  that  she  likes  to  see  at  the  beginning  what  will  be  covered  in 
whole  course.  This  was  not  visible  in  SS3011.  That  was  the  largest 
problem  for  her.  She  expects  to  be  self-paced  and  would  like  to  have  at 
least  a  complete  reading  list.  There  is  also  a  timing  issue  in  forums  and 
there  was  some  ‘just  in  time’  studying. 

•  He’d  like  more  multimedia  content,  video  stream  and  similar.  It  is  very 
bad  that  many  courses  in  NPS  do  not  use  any  computer  or  Internet 
capabilities,  3D  and  similar! 


Ninth  Question  -  Summary 

I  reduced  the  student  recommendations  to  their  key  messages  to  the  instructor 
(ranked  by  frequency): 

•  Fix  all  tech  problems  as  soon  as  possible  (three  students). 

•  Organize  occasional  meetings  with  students,  every  2  weeks  or  so  (three 
students). 

•  Provide  list  of  all  deliverables/obligations  at  the  beginning  (two  students). 

•  Introduce  some  real-time  or  combined  chat/forums  (two  students). 

•  Give  tougher  quizzes. 

•  Provide  more  feedback  from  the  instructor  (to  individuals  and  to  groups). 

•  Develop  a  backup  plan  (in  technical  sense). 

•  Provide  better  module  closeouts. 

•  Design  more  multimedia  and  graphical  course  content. 

•  Find  a  way  to  force  students  not  to  study  ‘just  in  time’ . 

•  Involve  external  subject  matter  experts. 

It  was  difficult  to  synthesize  these  categories  of  the  student  recommendations 
because  some  of  the  categories  overlap.  The  recommendations  address  the  following 
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issues:  course  web  design  -  four  responses;  technical  problems  -  four  responses; 
communication  between  the  students  and  the  instructor  —  four  responses;  general 
organization  of  the  course  -  three  response.  These  are  all  very  important  elements  in 
creating  a  positive  learning  environment  for  the  course. 

10.  Tenth  Question 

The  tenth  question  was:  ‘This  course  was  not  only  a  space-systems  course,  but 
also  a  new  experience  for  most  of  the  students  in  this  section.  Looking  at  the  course  from 
this  time  distance  and  do  you  think  you’ve  gained  some  other  benefits  besides  the 
course  objectives  themselves?  Please  comment.’ 

Here  are  the  student  comments  to  the  tenth  question: 

No 

•  No,  because  most  of  students  already  had  decent  computer  skills.  Future 
IS  going  towards  DL;  EXPOSURE  to  it  was  main  thing,  kind  of  trial-and- 
error. 

•  He  already  knows  everything  about  DL. 

•  Most  of  the  students  are  good  in  IT29. 

•  This  was  his  first  on-line  course,  but  he  works  a  lot  on-line.  Without  that 
knowledge,  it  would  be  harder.  So,  he  didn't  gain  anything  new. 

•  He  is  very  good  in  Web  and  PC  technology,  so  he  didn’t  gain  anything 
else.  Maybe  only  new  personal  experience,  and  he  may  do  something 
similar  in  the  future  (he  works  in  IT  area). 

•  He  had  a  plenty  of  on-line  learning  (10  courses)  already. 

•  Probably  not.  This  is  connected  with  issue  of  PC  proficiency  and  with 
experience  with  the  Internet.  He  is  very  good  in  that,  so  he  didn't  gain 
anything  else  significant. 

Only  writing 

•  For  him  personally,  only  challenge  is  to  write  better.  He  gained  nothing 
new  in  technical  sense;  he  is  very  proficient  in  web  technologies. 


29  Information  Technology. 
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Yes 


•  It  was  very  different  from  the  other  on-line  course  (IW)  she  had. 

•  She  liked  this  way  (of  learning)  and  would  like  more.  She  has  learned  that 
there  are  more  than  one  way  for  presenting  information,  and  PLENTY  of 
sources.  The  instructor  did  a  great  job. 

•  He  learned  a  lot  about  forums.  He  had  computer  proficiency  before. 

•  The  course  opened  for  him  doors  of  new  educational  opportunities,  and 

there  are  many  (Net-G  Navy  program  and  other  educational 
opportunities).  He  will  certainly  enter  into  such  programs,  and  that's  very 
good.  This  way  (DL),  educational/training  possibilities  can  be  much  better 
focused,  and  possibilities  and  availability  of  educational  opportunities  are 
increased  (and  he  needs  them).  In  DL,  it  is  also  possible  to  choose  only  the 
courses  one  needs. 

•  He  learned  that  there  is  still  a  long  way  to  go  to  a  good  learning  in  DL. 
People  are  different.  We  have  to  respect/recognize  well  subtle  aspects  in 
difference  between  DL  and  face-to-face  learning. 

•  He  learned  not  to  take  a  DL  course  if  he  is  not  ready  for  that. 

•  Before  coming  to  NPS,  he  hasn’t  used  the  Internet  much.  Now  he  is  much 
better  in  surfing,  searching,  solving  problems. 

•  It  additionally  opened  her  eyes,  enhanced  her  awareness  on  available 
options.  She  was  a  little  skeptical  about  learning  effectiveness  in  such 
course,  but  now  she  remembers  very  well  many  things  from  die  course  (2 
months  later).  She  is  very  interested  in  web  technologies.  It  can  be  a  very 
useful  learning  option,  if  it's  done  well. 

•  DL  is  the  future,  now  he  works  much  easier  within  it,  uses  it  without  being 
tensed,  but  he  still  doesn’t  have  enough  experience  to  compare  it  with 
other  on-line  courses.  A  course  like  this  one  can  provide  a  student  with 
better  understanding  of  future  educational/professional  opportunities. 


Tenth  Question  -  Summary 

The  responses  to  tenth  question  were:  ‘no’  -  seven  responses;  ‘only  writing’  -  one 

student  response;  ‘yes’  -  nine  responses.  Analysis  of  these  responses  shows  something 

very  interesting.  Practically  all  the  students  who  responded  ‘no’  used  one  key 

explanation:  they  are  already  very  experienced  in  IT  and  in  using  the  Web.  Quite 

differently,  the  students  who  responded  ‘yes’  had  very  different  reasons  for  their  answers: 
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new  learning  opportunities,  very  different  course  (from  other  on-line  course(s)),  enhanced 
awareness  on  options,  forums,  etc.  Only  one  of  these  responses  used  the  same 
explanation  as  the  students  who  answered  ‘no’.  He  explained  he  did  not  have  much 
experience  with  the  Internet.  In  other  words,  the  first  group  of  students  (‘no’  responders) 
were  more  focused  on  only  a  single  factor,  while  ‘yes’  responders  had  more  diverse 
reasons. 

11.  Eleventh  Question 

The  tenth  question  in  the  interviews  was:  ‘I  asked  you  many  questions  in  the  four 
on-line  questionnaires  and  in  this  interview.  If  you  think  there’s  anything  else  I  should 
know,  but  you  haven’t  comment  it  yet,  please  feel  free  to  do  it  now.’ 

Only  one  student  didn’t  respond.  Here  are  other  comments,  grouped  into  few 
typical  sets  of  responses: 


Technical  recommendations 

•  It  would  be  good  if  the  server  is  set  to  e-mail  replies  to  post  automatically 
to  original  post  writer’s  address.  The  instructor  was  the  key  to  the  success 
of  the  course  -  she  acted  exactly  in  accordance  with  tendencies  of 
decentralized  leadership  style. 

•  A  pilot-course  had  to  be  previously  done  in  order  to  clean  all  ‘bugs’.  There 
was  a  lot  of  confusion  with  passwords.  Too  many  questions  in  the 
beginning  connected  with  technical  problems,  and  if  somebody  is  getting 
‘bad  taste’  because  of  that,  that’s  bad  for  the  course.  Personally,  he  didn’t 
see  that  as  major  problem,  he  is  used  to  things  like  that,  especially  in  the 
military. 

•  It’s  certain  that  some  beginner’s  mistakes  will  not  happen  again.  It  all  had 
to  be  tested  better  (pages,  links),  instead  being  fixed  during  the  first  two 
weeks. 

•  Technical  problems  have  to  be  solved  before  the  beginning  of  the  course 
because  they  produce  resistance  in  students.  Fortunately,  most  of  the 
students  were  in  IT  area  and  they  are  familiar  with  such  problems.  In 
general,  it  functioned  well;  it  was  designed  intuitively. 
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Positive  opinions 


She  enjoyed  the  material.  In  on-line  course  it’s  easier  not  to  study 
regularly  -  slacking  off. 

It  was  pretty  decent  course,  and  nothing  important  was  lost  with  transfer  to 
DL  delivering. 

The  course  was  OK.  People  were  trying  hard,  and  he  personally  gave 
many  well-intentioned  recommendations  to  the  instructor.  The  students 
were  pretty  constrained  in  the  learning  process  (module  after  module). 
One  can’t  go  forward  in  advance,  same  as  in  ‘classical’  course.  One  can’t 
learn  at  one’s  own  pace. 


Other  advice,  comments,  and  a  question 

Skeptic:  On-line  is  more  time  consuming  -  that’s  the  key  point.  She  is 
pretty  skeptical  about  DL  while  on  duty,  even  if  it  gives  new 
opportunities.  It  all  depends  on  student,  how  much  time  he/she  invests  into 
a  course. 

More  real-time  events:  She  would  prefer  occasional  real-time  chat  as  an 
addition. 

Security:  Security,  log-ins  and  passwords  have  to  be  changed  (they 
shouldn’t  be  based  on  SSN).  It’s  privacy  issue! 

‘Switch’:  Many  students  said  they  haven’t  learned  much.  He  somewhat 
agrees,  but  that’s  because  they  compared  the  course  to  resident  courses. 
For  DL  one  has  to  make  a  ‘switch’  in  one’s  own  head.  It  is  possible  to  just 
give  minimum  comfort  in  the  course,  but  if  a  person  has  self-discipline 
and  makes  one’s  own  conclusions,  potentials  for  learning  are  great.  Links 
to  the  newest  information  is  an  advantage  for  DL;  books  do  not  have  that. 
The  textbook  was  very  good  too. 

‘Guinea  pigs’:  Face-to-face  courses  have  an  advantage  while  in  the 
school.  He  felt  that  his  class  was  kind  of  guinea  pigs,  and  that  produced 
resistance  in  many  students.  Everything  was  not  technologically  ready  for 
the  course. 

Quizzes  and  tests:  He  probably  gained  the  most  on  the  final  exam:  he  had 
to  collect  it  all,  to  organize,  to  formulate  as  a  recommendation  to  a  boss  - 
that  was  most  useful.  He  would  recommend  more  good  tests,  not  too  easy 
-  maybe  tougher,  could  be  e-mailed  to  students,  and  no  multiple  runs 
through  the  test. 

A  question:  He  asked  why  the  UCF  model  was  selected  for  application  in 
NPS. 


•  ‘Psychological  factors’:  On-line  learning  is  very  visual:  some  people 
don’t  like  to  learn  like  that.  Maybe  learning  style  should  be  tested  before 
sending  a  person  to  DL  course. 

Eleventh  Question  -  Summary 

The  final  comments  reflected  the  different  backgrounds  and  priorities  of  the 
students.  Some  addressed  technical  problems  in  the  course;  others  were  more  focused  on 
the  learning-environmental  or  psychological  factors  of  the  learning  process.  Their 
impressions  varied,  but  were  more  positive  than  negative.  Most  of  the  comments 
contained  useful  recommendations  and  suggestions  for  further  development  and 
improvements  of  the  course. 

12.  Interviews  With  The  Students  -  Conclusions 

The  interviews  were  individual,  semi-structured,  conducted  four  to  six  weeks  after 
the  course  was  completed,  and  only  with  students  who  volunteered.  Out  of  thirty, 
seventeen  students  agreed  to  give  interviews.  None  of  the  students  with  less  than  an  A- 
frnal  grade  were  among  them. 

Eleven  open-ended  questions  were  asked  in  the  interviews.  They  mostly 
addressed  some  important  issues  already  explored  through  the  on-line  questionnaires. 

When  asked  whether  they  learned  more  from  the  on-line  than  from  face-to-face 
course,  eleven  students  disagreed;  four  students  thought  the  learning  would  be 
approximately  the  same,  and  only  two  students  agreed.  Two  explanations  for  disagreeing 
were  dominant:  the  course  content  and  design,  and  student  motivation.  One  student 
mentioned  environment  as  a  key  reason.  The  students,  who  thought  of  the  learning 
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efficiency  as  the  same,  did  mention  differences  between  DL  and  face-to-face  courses,  but 
with  equal  opportunities  and  results. 

Asked  about  the  usefulness  of  the  forums,  sixteen  students  saw  them  as  ‘useful’, 
‘probably  useful’  or  ‘very  useful,  and  only  one  student  as  ‘not  useful.  Even  in  opposite 
responses,  the  same  argument  was  used  -  the  argument  that  the  forum  posts  represented 
the  student  opinions,  not  factual  material. 

Ten  students  saw  quizzes  as  useful,  and  six  students  as  not  useful.  It  seems  that 
the  difference  found  in  opinions  mostly  comes  from  a  different  interpretation  of  the  role 
of  the  quizzes  in  the  course.  Besides  that,  some  students  saw  quizzes  as  useful,  but 
anyway  did  not  like  them. 

Interactivity  issues  were  addressed  in  four  questions.  Predominantly,  students 

thought  the  amount  of  interaction  with  the  instructor  in  the  DL  environment  was  less  than 

in  the  face-to-face  classes  (six  said  -  ‘much  less’;  eight  -  ‘less’).  Analysis  of  then- 

responses  suggests  that  their  key  explanation  was  the  absence  of  the  instructor’s  personal 

presence.  Asked  about  the  quality  of  interaction  with  the  instructor,  students  most 

frequently  thought  it  was  good  (seven  students)  or  even  better  (two  students),  but  a 

substantial  number  of  students  had  the  opposite  opinion  (five  students  -  ‘less’).  Lack  of 

the  instructor’s  presence  was  again  the  main  explanation  in  the  latter  case.  Many  students 

who  thought  quality  of  interaction  was  good  or  better  mentioned  some  notable,  but  not 

overwhelming  problems  in  the  DL  course.  When  questioned  about  the  amount  of 

interaction  between  the  students,  eleven  students  responded  that  there  was  less  such 

interactivity  in  the  course  compared  to  a  face-to-face  course.  Those  who  though  there  was 

greater  interaction,  emphasized  their  explanations  that  students  were  forced  or  requested 

239 


to  post  to  forums  and  thus  to  interact  so  much.  Finally,  students  tended  to  see  the  quality 
of  the  interaction  between  students  better  than  in  a  face-to-face  class:  eight  said  ‘better’, 
one  -  ‘little  better’,  and  only  three  said  ‘worse’.  Several  students  though  the  quality  was 
enhanced  because  they  had  to  write  postings,  which  stayed  on  a  server  until  the  end  of  the 
course.  Some  added  that  forum  posts  serve  as  references  for  further  discussions  and, 
therefore,  the  quality  must  be  better. 

Most  students  would  think  carefully  about  whether  to  choose  an  on-line  or  face- 
to-face  course:  twelve  students  responded  it  depends  on  the  environment,  on  the  nature  of 
a  course,  and/or  on  the  instructor.  Only  five  students  answered  ‘face-to-face’,  but  most  of 
them  added  that  it  might  depend  on  the  environment  or  other  factors. 

The  students  also  gave  interesting  and  applicable  advice  and  recommendations. 
The  recommendations  were  dealing  with  fixing  the  technical  problems,  organizing 
occasional  meetings  with  the  students,  better  defining  what  is  expected  from  the  students 
(all  deliverables),  and  other  issues. 

In  justifying  their  responses  to  the  questions  on  the  course  as  a  new  experience 
that  includes  some  other  benefits  and  not  only  the  course  objectives,  students  responding 
‘no’  were  focused  on  fact  that  most  students  were  proficient  in  IT  technology,  while 
others,  who  responded  positively,  thought  about  several  factors. 

The  last  question  asked  the  students  to  add  other  comment  they  wish.  They 
responded  with  a  wide  spectrum  of  comments.  The  comments  were  mostly  forthcoming 
recommendations  for  improvements  of  the  course.  Diversity  of  their  responses  reflected 
student  backgrounds  and  priorities. 
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F.  INTERVIEWS  WITH  THE  INSTRUCTOR 

I  met  with  the  instructor  several  times  to  talk  about  different  issues  of  the  course.  I 

planned  four  sessions  exclusively  for  interviews  about  the  course.  The  questions  I 
intended  to  ask  at  a  particular  interview  were  sent  to  the  instructor  in  advance.  I  divided 
the  instructor’s  responses  by  the  main  issues: 

The  instructor’s  preparations  for  the  course 

As  soon  as  the  instructor  learned  she  had  to  design  and  deliver  the  course  in  on¬ 
line  environment,  she  started  to  organize  material  into  directories,  resources,  sources  of 
information,  what  took  approximately  several  (maybe  five)  hours  a  week,  for  three 
months.  From  the  end  of  January  to  March  2000  (8  weeks  in  total)  she  attended  an  EDL 
6543  (Interactive  Distributed  Learning  For  Technology-Mediated  Course  Delivery) 
course  at  the  University  of  Central  Florida.  The  course  emphasized  the  development  of 
web-based  learning  modules  and  on-line  learning  in  general.  During  the  course  she 
developed  a  test-module  (not  included  in  the  course  later).  From  April  to  June  2000  she 
devoted  60%  of  her  working  time  to  course  development. 

Commander  Higgins  did  not  have  any  previous  web-development  or  on-line 
teaching  experience.  She  has  no  previous  teaching/instructing  experience  at  all  except  for 
briefings,  group  workshops,  and  seminars. 

Preparations  for  an  on-line  course  were  a  very  creative  process  and  it  was  a  very 
positive  and  an  intensive  learning  experience  for  her. 

On  the  ‘UCF  model’ 

She  fully  supports  the  ‘UCF  model’.  She  believes  that  it  is  essential  to  enter  into 

the  transformation  process  and  to  ensure  there  is  a  supportive  infrastructure  for  the 
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faculty.  It  is  not  ‘mechanical’  transformation  into  different  media,  but  it  is  necessary  to 
also  start  thinking  in  terms  of  how  people  learn  and  how  best  to  use  available  tools  to 
accomplish  efficient  learning. 

The  UCF  model  is  process-focused  and  includes  continuous  revising  of  the 
materials  and  obtaining  feedback  from  students.  I  found  significant  similarities  between 
these  principles  of  the  UCF  approach,  the  explanations  the  instructor  gave  me,  and  the 
students  responses  about  the  course.  UCF  ensures  that  every  DL  instructor  collaborates 
with  am  instructional  designer  who  is  a  constant  link  between  the  instructor  and  several 
UCF  faculty  members. 


Initial  problems  in  the  course 

Scheduling  problem:  45  students  appeared  on  first  day,  because  of  the  mix-up  in 
the  Registrar’s  Office.  She  expected  only  25  students.  From  this  problem  another  one 
emerged  -  a  problem  with  logins  and  passwords. 

Second,  during  the  first  week  of  the  course  the  UCF  servers,  which  hosted  the 
course  on-line  materials,  did  not  function  because  of  relocation  to  new  offices.  NPS 
students  were  not  in  the  UCF  databases  of  students,  which  caused  additional  problems 
with  logons  and  passwords. 

Firewall  problem:  some  computers  in  NPS  were  configured  differently  than 
computers  at  the  UCF  campus,  so  unexpected  problems  with  the  firewall  appeared.  This 
was  not  discovered  before  the  beginning  of  the  course. 
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It  took  approximately  two  weeks  to  solve  the  problems  described  above.  During 
one  of  the  following  weekends,  the  UCF  servers  were  down,  but  during  remaining  time 
there  were  no  similar  problems. 

Some  of  these  technical  problems  were  generated  exclusively  because  of  the 
circumstance  that  the  course  server  was  located  on  the  UCF  campus.  This  course  was  one 
of  the  first  non-resident,  non-UCF  courses  hosted30  and  supported  by  UCF,  but  taught  at 
NPS. 


Key  lessons  learned  for  the  instructor,  in  light  of  the  initial  problems  are: 

•  Complete  all  materials  before  the  beginning  of  the  course 

•  Limit  size  of  the  group.  For  instructor-beginner,  it  should  not  be  over 
fifteen  to  twenty  students. 

•  Assistant  to  the  instructor,  mostly  for  technical  issues,  but  also  for  course 
design,  content  preparation,  teaching  issues,  etc.,  is  of  essential 
importance. 


.  Thinking  about  the  beginning  phase  of  the  course,  the  instructor  concluded  she 
did  not  use  the  best  approach:  she  set  the  stage  and  gave  the  roles  to  the  students,  and  did 
not  personally  engage  the  on-line  dynamics  of  the  class  as  much.  Because  of  problem 
solving  process  at  the  beginning  of  the  course,  the  first  week  was  an  excellent,  valuable 
experience  for  her.  Also,  she  needed  additional  advice  about  moderating  the  forum 
discussion,  which  she  got  from  UCF  faculty  and  colleagues.  In  essence,  the  process  of  the 
instructor’s  improvement  continued. 


30  i.e,.  UCF  server  hosted  all  on-line  course  materials. 
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Quizzes 


Quizzes  were  not  planned  to  be  included  in  the  final  grade,  but  as  exercises  that 
provided  important  feedback  to  the  students.  It  was  not  important  how  many  points  a 
student  received,  but  the  students  did  not  quite  understand  the  instructor’s  intention. 
Students  did  not  like  the  word  ‘quiz’,  so  the  instructor  decided  to  change  it  in  the  next 
iteration  of  the  course  to  a  ‘self-assessment’. 

The  instructor  was  not  aware  that  she  would  encounter  so  many  technical 
problems  with  the  quizzes.  She  was  aware  she  would  have  an  exceptionally  high  volume 
of  work  in  the  first  iteration  of  the  course.  Therefore,  she  did  not  work  on  further 
development  of  the  quizzes  during  the  course,  as  much  on  any  detailed  further 
development  of  the  quizzes.  She  decided  to  add  more  open-ended  question  to  the  quizzes. 

This  was  the  first  on-line  course  to  most  of  the  students 

The  instructor  was  well  aware  that  this  course  was  the  first  on-line  learning 
experience  for  almost  all  students.  This  fact  could  cause  significant  frustration  for  them. 
She  knew  that  this  fact  was  not  so  important,  but  still  was  a  significant  issue  for  the  total 
amount  of  concern,  but  still  she  had  to  deal  with  it.  Due  to  this  fact,  she  told  them  that  if 
the  number  of  problems  were  too  great  they  could  drop  the  on-line  section  and  enroll  in 
the  face-to-face  section  of  the  same  course.  For  various  reasons,  ten  students  decided  to 
do  so:  some  because  of  their  frustrations  and  some  without  any  specific  reasons.  After 
that  transfer,  thirty-three  students  remained  in  this  class. 

After  the  first  week  of  the  course,  comments  and  recommendations  of  the 
remaining  students  to  the  instructor  were  very  constructive  and  useful.  The  instructor 
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concluded  that  after  the  first  week  the  students  demonstrated  effective  learning  and 
proved  to  be  able  to  use  the  course  software. 

Grading  criteria 

The  instructor  had  a  grading  sheet  for  each  student.  Final  grades  were  based  on 
the  following: 

•  ‘Doing  everything’  in  sense  of  reading  module  materials  and  participating 
in  every  forum  on  a  regular  bases  Forums  -  initiative  and  responding 

•  Answering  the  quizzes  regularly  (number  of  points  in  quizzes  was  not  so 
important) 

•  Final  paper 

According  to  the  instructor’s  estimation,  the  first  two  criteria  consisted  of 
approximately  50%  of  the  final  grade,  and  the  final  paper  the  remaining  50%. 

For  grading  the  final  papers,  she  had  three  criteria: 

•  Physical  influences  on  space  systems 

•  Impacts  of  commercial  space  operations  to  DoD 

•  Impacts  of  policies  on  accessing  space  systems 

Good  elaboration  in  all  three  areas  in  the  final  paper  would  mean  an  A  grade  for 
the  final  paper  which  was  50%  of  the  final  grade. 

Since  this  thesis  research  questions  address  learning  effectiveness,  and  final 
grades  were  only  objective  indicator  of  learning  outcomes,  the  instructor’s  explanation  of 
grading  criteria  is  very  important  for  interpretation  of  the  results. 
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The  student  feedback  to  the  instructor 


The  instructor  thinks  that  students  had  different  overall  affective  impressions 
about  the  course  partially  because  they  were  ‘different  types  of  people  in  personal 
characteristics  relevant  to  one’s  adaptation  to  a  DL  environment’. 

Approximately  half  the  students  visited  the  instructor  in  her  office.  Most  of  them 
had  a  very  positive  opinion  of  the  course.  Very  often  they  stated  that  while  they  were 
resident  hJPS  students,  they  would  prefer  a  face-to-face  compared  to  the  on-line  course. 
However,  they  liked  experience  of  an  on-line  course.  Their  responses  in  SOF  were  not 
surprise  for  the  instructor,  because  she  already  received  large  amount  of  feedback  from 
students. 

The  initial  quality  of  student  inputs  in  the  forums  far  exceeded  the  instructor’s 
expectations.  The  forum  posts  were  very  challenging  to  the  instructor  because  she  had  to 
find  optimal  timing  and  way  to  moderate,  facilitate,  lead,  and  control  the  threaded 
discussions.  On  the  other  hand,  trying  to  have  complete  control  over  that  process  would 
not  be  good. 

Several  students  suggested  real-time  on-line  chats  or  similar  kind  of  interaction. 
Near  the  end  of  the  course,  their  suggestions  emphasized  more  the  need  for  additional 
classroom  sessions,  maybe  once  a  week  or  every  14  days.  The  instructor  felt  that  need 
also.  Such  change  would  make  this  course  a  hybrid,  not  entirely  web-based  course.  That 
could  not  be  acceptable  for  all  future  non-resident  students. 

The  instructor  saw  some  of  the  students  in  the  group  only  once  during  the  first 
session  before  the  on-line  course  actually  began.  However,  even  before  the  course  was 
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over,  the  next  quarter’s  students  started  to  ask  her  about  the  on-line  section  and  expressed 
significant  interest.  Another  valuable  source  of  feedback  were  SOFs. 

What  the  instructor  decided  to  improve  for  second  iteration  of  the  course 

Other  researchers  in  this  area  usually  think  that  an  instructor  ‘stabilizes’  in  the 
third  iteration  of  the  on-line  course  they  teach.  Based  on  experiences  from  the  first 
iteration  of  the  course,  this  instructor  intends  to: 

•  Increase  amount  of  interaction  with  the  students 

•  Reduce  technical  problems  as  much  as  possible 

•  Avoid  last  minute  content  preparations 

•  Rename  quizzes  to  ‘self-assessment’  and  improve  them 

Other  lessons  learned 

The  course  was  more  time  consuming  for  her  than  she  had  expected.  Probably 
more  unexpected  problems  appear  in  this  type  (on-line)  of  course  than  in  face-to-face 
courses.  The  first  iteration  of  an  on-line  course  should  not  have  many  students,  and  the 
schedule  should  not  be  too  tight  Especially  at  the  beginning  of  the  course,  the  instructor 
needs  to  have  more  reserve  time,  especially  in  terms  of  unexpected  technical  problems. 

The  instructor  said  her  emphasis  was  on  course  implementation,  not  on  course 
design,  which  includes  redesigning  after  every  good  suggestion  or  recommendation  from 
the  students.  This  would  not  be  rational. 

Student  perception  of  the  course  load  in  an  on-line  course  follows  different  logic 
than  in  face-to-face  courses,  and  an  instructor  has  to  be  careftd  about  this,  especially 
when  teaching  such  a  course  for  the  first  time.  Also,  e-mail  and  forum  communication 
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demands  additional  caution,  which  is  different  from  the  usual  personal,  verbal  and 
nonverbal  communication  between  an  instructor  and  students. 

Some  students  were  anxious  about  the  on-line  course  in  the  beginning.  She  had  a 
line  of  the  students  in  front  of  her  office. 

She  plans  to  prepare  more  multimedia  materials  for  the  next  iteration  of  the 
course,  but  not  many  because  of  the  bandwidth  problem.  Some  students  do  not  have  high¬ 
speed  connections. 

The  instructor  also  added  that  not  having  previous  face-to-face  teaching 
experience  might  be  an  advantage  for  her.  She  did  not  have  to  unlearn  face-to-face 
versions  of  the  course  and  face-to-face  methods  of  instruction  did  not  interfere  with  the 
on-line  teaching  methods. 

At  the  end,  the  instructor  added  that  the  policy  on  DL  at  NPS  is  changing  in  terms 
of  enabling  more  people  to  attend  DL  courses  in  the  Navy,  not  only  in  the  fleet  on 
carriers  and  ships,  but  also  on  shore.  Also,  the  Navy  scheduled  significant  improvements 
in  bandwidth  on  ships.  In  other  words,  DL  gained  increased  organizational  support  at 
NPS  and  in  the  Navy,  which  was  an  additional  acknowledgement  and  motivator  to  the 
instructor  of  this  course. 
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V.  DISCUSSION 


This  thesis  was  initiated  by  the  need  for  evaluation  of  one  of  the  first  on-line 
courses  taught  at  NPS.  That  need  was  not  only  a  consisting  part  of  ‘UCF  model’  of  DL, 
but  also  an  important  step  of  permanently  evolving  instructional  design  process,  and  a 
part  of  academic  and  organizational  culture  in  NPS  -  what  is  primarily  embodied  in 
mandatory  Student  Opinion  Forms. 

Studying  a  single  case  of  relatively  small  number  (thirty)  of  SS3011  on-line 
course  students,  I  collected  data  from  nine  main  different  sources:  four  on-line 
questionnaires,  SOFs,  student  on-line  tracking,  final  grades,  interviews  with  the  students, 
and  interviews  with  the  instructor.  Even  though  it  was  not  possible  to  provide 
sophisticated  level  of  data  analysis  (because  of  small  sample,  mostly  ordered  categorical 
data,  and  because  large  part  of  instruments  was  administered  anonymously),  many 
relations  can  be  discussed  and  conclusions  possible  to  drive. 

Before  integrating  all  findings  and  their  implications  and  answering  this  thesis 
research  questions,  I  will  discuss  results  of  each  one  of  the  data  sources  I  used  in  this 
research. 

The  First  on-line  questionnaire  (A-l  or  ‘The  Basics’),  administered  at  the 

beginning  of  the  course,  provided  a  quick  overview  about  the  students  (quarter,  service, 

pay  grade),  about  their  relevant  previous  experiences  (on-line  courses,  space-related 

courses),  computer  and  Internet  user  proficiency,  expectations,  and  concerns.  It  may  be 

said  that  as  first  of  four  on-line  questionnaires  it  had  secondary  role  of  preparing  the 

students  for  ‘more  serious’  questionnaires  to  come.  However,  insight  into  responses  and 
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into  time  the  students  needed  in  average  to  fill  up  a  questionnaire  on-line  confirmed  that 
they  had  no  problems  in  completing  them. 

Finding  that  the  students  attending  later  quarters  tend  to  visit  the  course  homepage 
fewer  times,  to  send  fewer  posts  to  forums  (what  includes  follow  up  posts  too)  can  be 
interpreted  with  important  fact  every  NPS  student  has  to  leam  -  with  the  need  for 
selection  of  information  and  for  time  management.  Very  likely,  more  ‘senior’  students 
tend  to  be  more  selective  and  more  careful  with  their  time,  so  this  tendency  seems 
reasonable. 

The  data  indicated  that  vast  majority  of  the  students  had  neither  previous 
experience  in  on-line  learning  nor  experience  in  space-related  courses.  That  is  to  say,  this 
case  study  is  about  first  application  of  an  on-line  course  in  Space  Systems  Academic 
Group,  NPS,  first  time  by  this  instructor,  and  as  first  DL  course  for  almost  all  of  these 
students.  Briefly,  this  was  a  pilot  project  of  NPS  course  transformed  to  DL  using  ‘UCF 
model’. 

Most  of  the  students  perceived  themselves  as  pretty  proficient  in  using  computers 
and  in  on-line  search  techniques  (what  was  expected,  having  on  mind  their  curriculum), 
and  most  of  them  expected  to  improve  their  learning  through  the  on-line  course. 
However,  those  with  lower  proficiency  level  and  with  lower  on-line  search  skills  had 
higher  expectations  for  improvements,  while  more  computer  proficient  students  tended  to 
read  larger  number  of  forum  articles.  According  to  these  relations,  it  is  doubtable  that  the 
question  on  expected  learning  improvement  closely  corresponds  to  ‘self-efficacy’, 
probably  because  of  formulation  of  the  question  (it  does  not  address  any  visible  criterion, 
but  only  unclear  term  ‘improve  learning’). 
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Self-perceived  on-line  search  technique  also  has  interesting  correlations,  first  - 
with  self-perceived  computer  proficiency,  and  second  -  with  number  of  different  course 
pages  visited.  The  students  more  skilled  in  searching  on-line  visited  fewer  course  pages 
probably  because  they  find  easier  what  they  were  looking  for;  they  select  information  and 
save  time. 

Open-ended  questions  to  this  and  other  questionnaires  provided  very  interesting 
qualitative  data,  which  I  analyzed  using  grouping  of  responses  by  meaning,  i.e.,  by 
message(s)  the  responses  were  stating.  In  First  questionnaire,  student  comments  (last 
item)  were  more  positive  than  negative,  with  few  statements  of  personal  preference  of 
traditional  classroom  and  few  complaints  about  large  forums. 

In  general,  First  questionnaire  provided  pretty  positive  picture  and  insight  into 
relation  between  expectations  and  self-perceived  proficiency  before  the  course. 

As  Suchan  and  Crawford  (1998)  recommended,  this  research  included  relevant 
measures  at  different  points  in  time  (i.e.,  at  different  time  points  within  the  course),  from 
orientation  phase  (First  questionnaire  and  partly  A-2  questionnaire)  to  the  period  after  the 
course  was  completed  (interviews  with  the  students).  This  way,  important  aspects  would 
not  be  missed.  Also,  I  collected  many  qualitative  data  using  on-line  questionnaires  and 
interviews.  The  instructor  of  SS3011  on-line  course  followed  many  other 
recommendations  the  authors  listed  in  their  paper. 

The  second  questionnaire  (A-2)  was  administered  during  second  and  third  week 
of  the  course,  after  the  students  got  clearer  impression  about  the  course.  It  was 
anonymous  and  had  26  items:  17  were  five-point  scales  and  the  remaining  were  open- 
ended  questions. 


251 


This  questionnaire  addressed  much  wider  spectrum  of  issues  than  the  first  one. 
Responses  on  questions  about  comfort  level  mostly  provided  very  desirable  distributions, 
but  on  several  issues  there  were  individual  students  that  had  some  problems:  four 
students  were  ‘somewhat  uncomfortable’  in  setting  their  systems;  one  student  ‘extremely 
uncomfortable’  in  accessing  the  Internet;  two  students  uncomfortable  (one  ‘extremely’ 
and  one  ‘somewhat’)  in  word  processing;  four  students  uncomfortable  in  posting  to 
forums,  etc.  However,  knowing  for  significant  organizational  and  technical  problems  at 
the  beginning  of  the  course  (reported  by  the  instructor  in  the  interviews),  such 
distributions  of  responses  were  surprisingly  positive. 

Largely  neutral  responses  to  questions  about  comfort  with  technical  support  and 
help  indicated  that  many  students  didn’t  need  to  use  that  help  or  were  not  aware  of  that 
option. 

Their  responses  to  open-ended  question  about  needed  technical  help  were  mostly 
about  posting  to  the  forums,  what  was  currently  central  technical  problem. 

Interpretation  of  the  student  responses  to  eight  statements  about  comfort  level 
with  use31  of  different  features  in  the  course  (forums,  e-mail,  quiz,  links,  course 
overview,  etc.)  is  not  quite  clear,  because  the  question  is  whether  these  items  represent 
student  comfort  level  or  effectiveness  of  respective  features  within  the  course.  It  depends, 
I  would  add,  on  how  we  define  effectiveness.  Of  course,  the  student  responses  cannot  be 
objective  indicators,  but  their  subjective  views  of  supposed  effectiveness. 


3'  These  questions  were  targeting  the  way  how  these  features  supported  learning  process  in  the  course, 
not  how  the  students  felt  comfortable  using  them. 
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Analysis  of  correlations  between  the  variables  in  this  instrument  showed  that 
questions  asking  abut  comfort  levels  with  IT32  activities  and  on-line  activities  present  in 
the  course  (first  five  questions  in  the  instrument),  and  the  question  about  comfort  levels 
with  features  used  in  the  course,  do  correlate  with  at  least  six,  but  mostly  with  ten  or 
more  other  variables.  Several  correlations  were  over  0.70.  Four  questions  asking  about 
comfort  level  with  different  forms  of  technical  support  had  much  fewer  and  in  average 
much  lower  correlations  -  ranging  from  one  to  four  statistically  significant  correlations 
with  other  variables.  The  interpretation  is  much  harder  in  this  case.  Many  responses  were 
neutral  for  simple  reason  that  the  students  have  not  used  or  were  not  aware  of  possibility 
of  such  support. 

Very  interesting  were  responses  to  the  question  asking  them  what  they  liked  about 
the  on-line  course.  I  grouped  nineteen  responses  as  ‘time  managements  flexibility  and 
accessibility’,  what  is  very  similar  to  findings  of  Moskal  and  Dziuban  (2000). 

I  grouped  students  responses  to  the  question  about  their  concerns  to:  technical 
problems;  uncertainty  about  comparability  of  DL  to  face-to-face  courses;  learning  pace 
and  organization;  grading;  and  miscellaneous.  This  list  of  their  concerns  exceeds  main 
problems  the  instructor  outlined  in  interviews,  what  points  to  the  benefits  of  this  sort  of 
feedback  collection  from  DL  students.  Their  responses  reflected  not  only  current 
problems  they  were  facing  in  the  course,  but  also  issues  every  student  meets  -  issues  of 
self-organization  and  self-discipline  (‘learning  pace’)  put  into  DL  context. 

Student  responses  to  the  question  asking  them  how  would  they  build  and  teach  an 
on-line  course  were  mostly  recommendations  for  improvements  of  instructional  methods 

32  Information  technology. 
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or  for  fixing  technical  problems.  However,  six  students  would  not  change  anything  from 
this  course. 

Module  14  questionnaire  was  administered  close  to  the  end  of  the  course,  asking 
the  students  three  sorts  of  questions:  how  well  the  module  14  learning  objectives  were 
met;  comfort  level  with  the  course  features  and  relevant  technical  issues;  and  one  open- 
ended  question,  asking  for  additional  thoughts. 

First  seven  items  gave  positive  picture  on  module  effectiveness  in  terms  of  met 
learning  objectives.  Only  exception  was  the  question  on  interaction  with  a  subject  matter 
expert  (what  made  this  module  exceptional),  because  the  number  of  neutral  responses 
was  very  large  compared  to  other  six  questions  in  that  part.  Possible  explanation  is 
students  who  chose  neutral  answers  probably  missed  the  opportunity  to  interact  with  the 
expert. 

Responses  to  these  seven  statements  significantly  correlated  with  other  variables 
from  the  instrument,  but  some  also  with  some  of  the  on-line  student  tracking  indicators 
and  few  variables  in  First  questionnaire. 

The  second  group  of  questions  was  parallel  to  identical  questions  in  Second 

questionnaire,  so  comparison  of  same  variables  in  different  time  points  in  the  course  is 

made.  Distributions  were  very  similar,  showing  some  improvements  over  the  course. 

Most  of  them  had  several  significant  correlations,  mostly  with  other  question  from  this 

group  and  with  on-line  tracking  indicators.  Very  interestingly,  two  questions  (about 

comfort  level  with  quiz  feature  used  in  the  course  and  about  comfort  level  with  word 

processing)  correlated  with  final  grades  too.  All  on-line  tracking  indicators  had 

significant  had  significant  correlations  with  other  variables,  some  with  several  variables 
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from  different  instruments,  what  undoubtedly  confirmed  they  have  to  be  considered  as 
important  data  source  in  DL  effectiveness  studies. 

Comments  added  at  the  end  of  this  questionnaire  were  mostly  positive  and 
constructive,  but  there  were  some  hybrid  comments,  some  complex,  and  some  skeptical. 

Final  survey,  the  last  of  four  on-line  questionnaires  in  this  research,  were 
administered  after  the  completion  of  the  course,  before  the  students  got  their  final  grades 
-  so  possible  impact  of  grades  to  student  responses  was  avoided. 

First  six  questions  followed  the  course  learning  objectives  and  most  of  the 
students  responded  positively.  Only  the  question  asking  about  usefulness  of  final  exam  in 
terms  of  helping  a  student  to  synthesize  the  concepts  learned  in  the  course  had  four 
negative  responses.  However,  what  is  especially  intriguing  with  these  questions  are 
correlations:  most  of  significant  correlations  seemed  to  be  in  average  most  numerous  and 
highest  with  other  of  these  six  questions,  but  that  was  not  absolute  rule.  Correlations  with 
some  other  questions  were  also  significant,  e.g.  with  questions  on  flexibility,  questions  on 
quality  of  interaction  with  the  instructor,  questions  on  usefulness  of  some  of  the  features 
used  in  the  course,  or  question  on  liking  some  of  the  course  on-line  features  and 
activities.  These  correlations  may  seem  irrelevant  if  one  reviews  theories  mentioned  in 
first  two  chapters,  but  they  may  raise  new  question  and  new,  unexpected  constructs  in 
description  and  analysis  of  DL  process.  They  are  statistically  significant  and  they  seem  to 
discover  new  relations,  for  which  we  still  don’t  have  theoretical  terminology. 

Seven  next  questions  asked  the  students  about  usefulness  of  the  following  course 

features  or  elements:  guidelines  (two  questions);  textbook;  on-line  module  readings;  the 

links  to  websites;  forum  exercises;  and  quizzes.  The  distributions  of  these  variables  were 
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strongly  positive,  but  some  of  them  had  notable  number  of  negative  responses  too,  those 
addressing  usefulness  of  forums  and  quizzes.  This  was  another  confirmation  that  the 
students  significantly  differ  in  their  interpretation  and  perception  of  forums  and  quizzes 
more  than  in  interpretation  and  perception  of  other  course  features.  Most  of  these  seven 
variables  had  very  few  significant  correlations:  from  one  to  four.  Only  exception  was  the 
question  about  usefulness  of  module  readings,  with  ten  significant  correlations.  Again, 
some  of  correlations  were  not  expected  and  are  hard  to  explain. 

Liking  of  a  course  feature  correlates  with  an  opinion  on  its  usefulness,  but  it  is  not 
the  same.  For  instance,  correlation  between  the  responses  about  liking  the  forums 
assignments  and  the  responses  about  usefulness  of  forum  exercises  was  p  =  0.62; 
correlation  between  responses  about  liking  the  textbook  readings  and  responses  about 
usefulness  of  the  textbook  was  p  =  0.64;  the  same  for  module  readings  was  p  =  0.54;  and 
for  website  links  p  =  0.56.  These  are  statistically  significant  but  mild  correlations.  In 
cases  of  forum  exercises/assignments  and  the  textbook,  these  correlations  were  highest 
among  very  few  other  significant  correlations,  but  for  module  readings  and  website  links 
they  were  not.  Moreover,  in  latter  cases  some  other,  unexpected  correlations  appeared  to 
be  higher,  and  there  were  much  more  significant  correlations.  The  question  asking  about 
liking  the  module  readings  had  ten,  and  the  question  asking  about  liking  website  links 
had  nine  significant  correlations  with  other  items  in  the  instrument.  Interpretation  of  this 
difference  in  numbers  of  correlations  would  be  that  latter  two  correlations  (module 
readings  and  links)  were  more  interrelated  to  many  other  factors  of  the  course  learning 
process,  while  examples  like  textbook  of  forum  exercise  represent  more  specific, 
relatively  isolated  sort  of  activities  within  the  course  context  (the  textbook  is  not  on-line 
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feature,  but  is  very  important  element  of  the  learning  process,  while  website  links  is  the 
course  material  design  and  content  feature).  Among  others,  responses  about  some  of 
aspects  of  interactivity  (amount  of  interaction  with  other  students  and  quality  of 
interaction  with  the  instructor)  were  also  significantly  correlated  with  two  described 
features  (module  readings  and  website  links). 

The  student  responses  to  open-ended  questions  were  very  constructive  and 
provided  interesting  recommendations.  They  had  positive  opinions  on  instructional 
strategies  used  in  the  course,  but  most  of  them  didn’t  think  the  course  fundamentally 
changed  their  approach  to  learning. 

From  responses  to  A-2  questionnaire  we  found  that  the  students  saw  the  flexibility 
(in  sense  of  time  management)  to  be  very  be  beneficial.  In  Final  survey  the  research  went 
one  step  further  and  added  two  questions  to  determine  whether  that  flexibility  was 
beneficial  only  within  the  course  or  in  terms  of  the  student  success  in  other  courses.  The 
student  responses  confirmed  that  the  latter  was  more  significant  -  the  time  flexibility  was 
beneficial  to  this  sample  of  students  mostly  in  order  to  succeed  in  other  courses. 

The  interactivity  issues  were  among  the  most  interesting  points  in  this  research. 
The  student  responses  in  Final  survey  showed  that  most  of  the  students  thought  that 
amount  of  interactivity  between  the  students  and  the  instructor  was  less  (compared  to  a 
face-to-face  course).  There  was  a  similar  trend  concerning  quality  of  interaction  between 
the  students  and  the  instructor,  but  number  of  neutral  responses  clearly  dominated. 
Similarly,  the  student  responses  on  amount  and  quality  of  interaction  between  the 
students  indicated  decrease,  with  significant  number  of  neutral  responses  and  with  few 

opposite  opinions.  This  does  not  match  findings  of  some  other  researchers.  Wang, 
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Dziuban,  and  Moskal  (2000)  found  that  UCF  students  saw  quality  of  interaction  with  the 
instructor  increased  in  on-line  courses,  but  findings  about  quantity  were  not  so  clear. 
Interactions  with  other  students  were  perceived  as  slightly  increased  in  quality,  but  it  was 
not  clear  about  the  quantity  (what  also  partially  depends  on  way  how  we  ask  the  students 
about  their  opinion  -  on-line  or  in  mailed  survey).  According  to  Hartman,  Dziuban,  and 
Moskal  (1999)  and  Moskal  and  Dziuban  (2000),  vast  majority  of  faculty  saw  the  amount 
and  the  quality  of  interaction  significantly  increased  in  comparison  to  face-to-face 
courses,  especially  in  cases  of  fully  on-line  courses.  However,  the  authors  (Moskal  and 
Dziuban,  p.  26)  anticipated  that  relation  knowing  that  the  faculty  was  requiring  students 
to  post  to  the  forums,  i.e.,  they  were  forcing  students  to  interact.  Similar  was  in  the  case 
analyzed  in  this  thesis,  and  some  of  the  student  comments  gave  the  same  explanation. 

The  student  responses  to  Long-Dziuban  Reactive  Behavior  Protocol  were 
surprising  if  compared  to  findings  of  other  authors  (Dziuban,  Moskal,  and  Dziuban 
[1999]).  In  case  of  sample  in  this  thesis  research,  much  more  students  (50%)  belonged  to 
Passive  Independent  type  (compared  to  12%  of  UCF  students  in  mentioned  study). 
However,  supposed  differences  between  two  populations,  selectivity,  environment,  and 
other  possible  factors  do  not  allow  conclusions  without  additional  and  more  focused 
research.  Same  holds  for  three  found  significant  correlations. 

Student  Opinion  Forms,  administered  on  official  NPS  forms  at  the  end  of  the 
course,  also  provided  interesting  information.  Statistical  analysis  of  responses  was  limited 
because  of  ‘No  comment’  responses,  which  had  to  be  excluded  from  calculations. 
Besides  that,  two  questions  seem  not  to  be  appropriate  for  DL  environment  and  NPS 
should  make  changes  in  SOF  forms  to  eliminate  this  problem. 
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Finding  from  First  questionnaire  about  little  importance  of  whether  a  student 
attend  first  or  sixth  quarter  was  confirmed  in  SOFs  -  this  item  did  not  have  any 
significant  correlations  with  other  variables  in  this  instrument.  Same  holds  for  number  of 
credit  hours. 

All  distributions  were  dominantly  positive  about  different  aspects  of  the  course 
(respectively),  with  sporadic  negative  or  neutral  responses.  Large  number  of  correlations 
between  the  responses  to  SOF  questions/statements  was  statistically  significant,  but  they 
were  reflection  of  targeted  constructs  the  instrument  was  intended  to  assess.  I  would  add 
that  this  thesis  and  other  relevant  follow-up  theses  and  research  might  provide 
suggestions  and  recommendations  what  could  be  added,  removed  or  changed  in  SOFs  for 
DL  courses  to  serve  their  purpose  even  better. 

Some  of  the  correlations  provided  additional  in-depth  insights  into  relations 
within  DL  learning  context,  like  positive  correlation  between  the  student  views  of  the 
instructor’s  course  objectives  made  clear  and  the  students  feeling  stimulated  in  the 
subject  area.  Such  findings  are  closer  to  general  pedagogical  issues  than  to  DL  specifics. 

Student  on-line  tracking  data  were  one  of  surprises  in  this  research.  All  tracking 
indicators  had  significant  correlations  with  other  variables:  with  other  tracking  indicators; 
with  variables  from  First  questionnaire,  Module  14  questionnaire;  and  with  final  grades. 

For  most  of  the  indicators,  correlations  with  comfort  level  with  quiz  feature, 
posting  to  forum,  accessing  the  Internet,  and  with  some  other  variables,  were 
exceptionally  high.  The  instructor  especially  appreciated  the  student  originality  and 
initiatives  in  forums  postings,  what  was  proved  by  correlation  between  the  number  of 
original  posts  and  the  final  grades. 
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The  distributions  of  total  number  of  hits  or  number  of  forum  postings  had  wide 
ranges  between  individual  minimums  and  maximums.  There  were  over  500  postings  in 
forums,  and  their  length  varied  from  just  a  few  lines  to  several  paragraphs  of  text,  so  it 
was  not  easy  to  synthesize  their  content.  Also,  brief  analysis  of  dates  and  time  of  posting 
the  forums  messages  by  time  in  day  and  by  day  in  week  confirmed  convenience  and 
flexibility  of  DL  course.  Unfortunately,  in  this  research  it  was  not  possible  to  examine 
relations  of  on-line  tracking  indicators  with  some  other  variables,  like  Long-Dziuban’s 
reactive  behavior  types,  what  is  one  of  possible  directions  for  future  research. 

Conclusion  from  these  findings  about  on-line  tracking  is  clear:  on-line  tracking 
indicators  are  relevant  in  DL  evaluation  and  can  significantly  contribute  not  only  in 
evaluation  efforts  but  also  in  further  research  of  DL  effectiveness  factors. 

The  instructor  stated  in  one  the  interviews  that  the  student  on-line  activities  were 
one  of  criteria  for  final  grade.  This  is  confirmed  with  significant  correlations  between 
final  grades  and  all  on-line  tracking  indicators.  Final  grades  also  had  two  other  significant 
correlations,  with  comfort  levels  in  quizzes  and  with  comfort  level  in  word  processing. 
However,  methodological  limitations  because  of  small  sample  and  non-normal 
distributions  indicate  need  for  more  thorough  research,  with  large  sample. 

Interviews  with  the  students  were  conducted  for  several  reasons:  first,  they 
provided  more  qualitative  data;  second,  it  was  possible  to  conduct  them  after  the  course, 
when  the  students  were  free  of  final  exams  pressure,  and  when  they  had  substantial  time 
distance  from  completed  SS301 1  course;  third,  my  choice  of  the  interview  questions  was 
inspired  by  the  most  controversial  items  from  all  four  on-line  questionnaires,  by  the 
research  questions,  and  by  the  instructor’s  recommendations. 
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The  most  of  the  student  did  not  think  they  learned  more  from  on-line  course  than 
they  would  learn  from  a  face-to-face  course.  However,  they  saw  forums  as  useful  and 
quizzes  as  relatively  useful,  and  it  seems  that  die  differences  in  opinions  occurred  mostly 
because  of  different  interpretation  of  the  role  of  these  features  in  the  course. 

Asked  to  compare  the  interactivity  in  the  DL  course  with  interactivity  in  face-to- 
face  courses,  they  thought  the  amount  of  interaction  with  the  instructor  decreased,  but  the 
quality  relatively  increased  or  was  the  same.  Similarly,  most  of  them  thought  that  amount 
of  interaction  between  the  students  decreased  in  the  on-line  course,  but  the  quality 
increased. 

In  a  situation  to  choose,  most  of  the  students  would  consider  taking  on-line 
instead  of  a  face-to-face  course,  but  their  decision  would  depend  on  the  environment,  the 
nature  of  a  course,  or  on  the  instructor. 

At  the  end  of  the  interviews,  the  students  provided  many  interesting  comments 
and  recommendations,  which  predominantly  reflected  their  backgrounds  and  priorities. 

The  instructor  provided  many  important  statements  about:  her  preparations  fpr  the 
course;  her  view  of  the  ‘UCF  model  of  DL’;  initial  problems  in  the  course;  the  grading 
criteria;  the  student  feedback;  needs  for  improvements  in  the  course;  and  lessons  learned. 
Her  comments  and  explanations  put  new  light  on  several  issues  present  in  the  student 
responses  in  the  questionnaires  and  the  interviews,  and  without  this  part  of  the  research 
many  wrong  conclusions  could  be  made. 

Having  on  mind  all  organizational  and  technical  problems  at  the  beginning  of  the 
course  she  described  (exceptionally  large  number  of  the  students  appearing  on  first  class, 
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firewall  problems,  software  ‘bugs’,  problems  with  UCF  server  being  down,  etc.),  many 
negative  student  responses  do  not  seem  to  address  the  curse  itself,  but  the  described 
circumstantial  problems.  However,  there  were  also  many  positive  reactions  from  the 
students  expressed  in  different  occasions  but  not  explicitly  visible  in  the  data  collected  in 
this  research. 

Teaching  the  on-line  course  for  this  instructor  was  creative,  demanding,  and 
continuous  learning  process,  and  she  provided  several  lessons  learned  during  the  course, 
which  are  directed  to  some  organizational  issues,  the  course  design  and  content  issues, 
feedback  to  the  students,  etc. 

The  most  astonishing  fact  from  the  interview  with  the  instructor  was  that  she  did 
not  have  any  previous  experience  in  on-line  teaching,  but  most  of  the  data  from  this 
research  confirmed  that  the  course  in  general  succeeded,  in  spite  of  this  fact  and  all  other 
problems  I  mentioned.  Many  experienced  DL  instructors  assess  that  most  often  for  a  DL 
instructor  is  necessary  to  go  through  three  iterations  of  teaching  a  DL  course  to  become 
much  more  comfortable,  stabile,  and  successful.  Organizational  support  is  also  important 
for  an  instructor’s  success,  what  also  takes  time  and  efforts  in  organizations  where  DL 
education  is  still  not  part  of  culture. 

Looking  back  to  the  theoretical  models  presented  in  introductory  chapter  and  in 
literature  review,  I  must  emphasize  that  the  case  analysis  results  cannot  directly  provide 
confirmation  or  results  denying  such  models.  Case  study  made  on  small  sample  and  in 
specific  circumstances  cannot  ensure  firm  data  that  could  be  generalized  to  such  a  model. 
However,  quantitative  and  qualitative  analysis  confirmed  that  several  groups  of  factors 
could  be  identified  on  the  bases  of  their  relations  and  qualitative  analyses. 
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Learner’s  characteristics  from  Baldwin  and  Ford’s  model  of  learning  design  are 
certainly  present  and  very  important  group  of  factors,  what  has  support  from  various  data 
in  this  research.  One  of  factors  in  this  group  that  should  be  considered  in  future  research 
is  Log’s  theory  of  reactive  behavior  patterns,  but  there  are  some  other  theories  (like 
theory  of  learning  styles)  and  constructs  (efficacy,  organizational  commitment)  that  may 
play  important  role  in  DL  effectiveness.  Students  varied  by  motivations  and  expectations. 
The  research  confirmed  that  more  computer  proficient  students  could  have  lower 
expectations  to  gain  from  a  DL  course.  However,  the  responses  depend  on  what  has 
really  been  asked.  In  this  research,  the  question  about  student  expectations  asked  about 
improvement  in  learning,  not  general  expectation. 

Learning  design  was  also  addressed  in  large  number  of  research  items.  The 
forums,  website  links,  module  objectives,  guidelines,  and  numerous  other  design 
elements  were  examined.  The  data  confirmed  that  there  are  many  complex 
interrelationships  between  these  and  other  issues  during  the  course,  including  some 
unexpected  relations,  not  known  from  existing  literature.  Qualitative  analysis  added  more 
issues  to  be  considered  in  future  DL  courses. 

Work  environment  factors  in  this  case  were  more  visible  from  the  instructor’s 
than  from  the  student  side.  There  are  many  new  initiatives  in  U.S.  Navy  supporting  DL 
training  and  education,  but  that  issue  was  not  directly  addressed  in  the  research.  The 
majority  of  students  didn’t  know  would  their  future  duties  include  area  covered  by  this 
course,  so  examining  the  role  of  opportunity  to  use  the  learned  from  the  course  didn’t 
have  much  sense. 
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One  of  the  findings  in  this  research  is  probably  the  key  role  of  interactivity,  which 
in  fact  means  feedback  loops  from  the  instructor  to  the  students  and  vice  versa,  but  also 
between  the  students.  Role  of  the  interactivity/feedback  is  mentioned  in  many  other 
articles  and  addressed  in  several  researches,  but  is  not  visible  in  these  models. 

Kraiger’s  classification  of  learning  outcomes  had  indirect  support  in  the  data:  the 
research  found  that  liking  a  course  feature  or  an  element  of  the  course  content  (affective 
outcome)  and  being  aware  (i.e.,  know,  be  convinced)  of  its  usefulness  (cognitive 
outcome)  is  not  the  same,  but  correlates.  Improvements  in  using  on-line  activities 
(posting  to  forums,  using  links, etc.)  represented  skill-based  outcomes,  but  not  in  terms  of 
the  course  objectives,  only  in  terms  of  skills  necessary  to  successfully  learn  in  DL 
environment. 

The  analysis  of  the  student  responses  to  questions  about  meeting  the  course 
objectives  and  of  final  grades  (which  was  only  objective  measure  of  cognitive  outcomes 
and  in  fact  of  course  effectiveness)  showed  very  complex  relationships.  Two  sorts  of  data 
appeared  to  be  most  correlated  with  the  grades  and  with  the  student  responses  about 
course  objectives:  on-line  tracking  indicators,  and  some  ‘general  technical  skills’ 
indicators  (comfort  level  in  using  forums,  in  sending  e-mail,  etc.).Only  conclusions  I  can 
make 

I  would  conclude  that  Belanger’s  framework  for  DL  was  closest  to  the  reality  of 
learning  process  within  this  on-line  course.  This  model  assumes  bi-directional  relations 
between  the  ‘players’  in  the  framework,  I  would  suggest  that  crucial  role  of 
interactivity/feedback  be  explicitly  stated  in  the  model.  Society  impact  and  learning 

outcomes  desired  by  the  society  are  implicitly  present  in  the  case  I  examined,  but  there  is 
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no  need  to  comment  it.  All  three  sides  -  the  learners,  the  instructor,  and  the  institution 
(here  the  large  part  of  institution’s  role  was  covered  by  UCF)  -  were  present  and 
interacted  during  the  course. 
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VL  CONCLUSIONS 


In  spite  of  the  fact  that  this  course  was  the  first  taught  by  the  instructor  and  in 
spite  of  organizational  and  technical  problems  at  the  beginning  of  the  course,  the  web- 
based  SS301 1  course  design  met  desired  learning  outcomes. 


The  student  expectations  and  motivation  were  predominantly  positive,  but  not  in 
consensus:  small  students  were  skeptical,  and  would  rather  attend  a  face-to-face  course. 
On-line  reactive  behavior  patterns  found  in  this  sample  significantly  differed  from  other 
studies,  but  there  is  no  enough  support  in  data  for  firm  interpretation.  The  student 
affective  reactions  were  predominantly  positive.  Utility  reactions  correlated  with 
affective,  but  not  strongly.  Only  slight  positive  changes  over  the  course  were  detected. 


Main  lessons  learned  from  this  course  are: 

•  Technical  problems  should  be  fixed  as  soon  as  possible,  best  before  the 
course. 

•  The  forums  should  be  limited  by  number  of  participants  and  by  size  if 
posts. 

•  Feedback  from  an  instructor  is  very  important,  but  the  other  dimensions  of 
interactivity  (students  to  instructor;  between  the  students)  are  also  crucial 
for  the  success  of  an  on-line  course. 

•  An  on-line  course  can  provide  desirable  flexibility  and  convenience  of  use 
to  resident  students,  but  it  would  be  more  effective  in  non-resident 
environment. 

•  A  course  evaluation  is  an  integral,  important  part  of  DL  education.  It 
should  be  conducted  in  different  time  points  of  the  course,  using 
qualitative,  not  only  quantitative  methods  too. 

•  For  an  instructor,  especially  if  it’s  his/her  first  DL  teaching  experience, 
DL  course  is  very  time-consuming,  but  creative  process  with  lots  of 
learning. 
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Other  important  findings  of  this  research: 


The  course  organization  was  endangered  by  major  scheduling  problem  at 
the  beginning,  but  later  it  went  on  successfully. 

The  pedagogy  used  in  the  course  was  very  effective  for  most  students,  but 
small  number  of  students  had  problems.  Long’s  typology  or  some  other 
approaches  to  individual  differences  relevant  for  DL  learning  could  be 
used  as  helping  tool  in  providing  such  students  an  adequate  instructional 
counseling  or  other  needed  support. 

The  technology  used  in  this  course  had  problems  at  the  beginning,  what 
underlines  the  need  for  adequate  technical  and  organizational  support  to  a 
DL  instructor.  Technology  should  be  ‘invisible’,  so  the  learning  process 
would  be  more  efficient  and  attractive  to  students  and  faculty. 

This  research  opened  many  possible  directions  for  future  research.  Many 
correlations  between  the  variables,  trends,  and  other  indications  found  in 
this  research  cannot  be  interpreted;  some  of  the  existing  constructs  should 
be  reexamined,  and  may  be  some  new  created.  The  most  important  is  to 
understand  the  complexity  and  interdependence  of  many  factors  relevant 
for  effectiveness  of  a  DL  course. 
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APPENDIX  A.  A-l  QUESTIONNAIRE 

NUMBER  OF  QUESTIONS:  25 


THE  BASICS 

Your  Demographics  and  Studying  Strategies 

The  on-line  Distributed  Learning  (DL)  environment  of  SS3011,  Space  Systems 
Technology  and  Applications,  is  new  to  DoD  and  Naval  Postgraduate  School.  We  would 
like  to  gather  data  through  the  course  that  will  assist  us  in  improving  future  DL  courses. 

In  an  on-line  course-learning  environment,  teaching  strategies,  in  particular,  are 
implemented  in  a  manner  that  will  enable  students  to  take  an  active  role  in  their  learning. 
To  achieve  this  goal,  courseware  developers  have  begun  to  think  about  ways  to  foster 
’learner-centered’  on-line  instruction.  This  process  has  included  not  only  looking  at 
demographics  of  the  adult  learner,  but  also  the  professional/career  goals  and  objectives  as 
well  as  the  studying  strategies  of  those  taking  electronic  courses.  These  types  of  data  can 
assist  courseware  developers  and  researchers  to  ensure  that,  ‘teaching  practices’  are 
responsive  to  the  needs  of  the  on-line  learner.  Your  help  in  responding  to  the  following 
questions  will  aid  us  in  this  endeavor. 

Question  1 

Your  e-mail  address: 


Question  2 

What  is  your  student  status  for  this  course? 
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c  1.  Full-time  Student  at  NPS. 

c  2.  Full-time  Military,  Auditing  Course, 

r  3.  Other. 

Question  3 

If  you  answered  ‘other’  to  the  prior  question,  please  enter  your  status  here: 


Question  4 

About  your  academic  and  professional  careers. . . 

What  is  your  Service? 

r  1.  USN 
r  2.  USMC 
r  3.  USA 
r  4.  USAF 

r  5.  OTHER  -  complete  the  following  question. 


Question  5 

In  the  answer  to  the  previous  questions  was  ‘other’,  please  fill  in  your  service  here. 


Question  6 

What  is  your  pay  grade? 


r  1.  01 

r  2.  02 

r  3.  03 

r  4.  04 

r  5.  05 

r  6.  Other 


Question  7 

What  is  your  Designator/MOS? 


Question  8 

Including  this  quarter,  how  many  academic  quarters  have  you  been  enrolled  at  NPS? 

Answer: 


Question  9 

What  is  your  curriculum? 

Answer: 
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Question  11 

Briefly  list  your  previous  duty  stations  and  billets. 


Question  12 

What  is  your  next  duty  of  assignment  (if  known)? 

Answer: 


Question  13 
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List  prior  space  system-related  tours  (e.g.,  commands/jobs). 


Question  14 

List  previous  university-level  space  systems-related  courses. 


Question  15 

About  your  experience,  to  date,  in  taking  an  on-line  course. . . 
Title  of  on-line  course  you  are  enrolled  in  for  this  survey: 


r  1 .  SS  301 1  Space  Systems  -  Technology  and  Applications. 

Question  16 

Have  you  experienced  an  on-line  course  (or  courses)  that  used  a  web  site  to  do/complete 
coursework? 

r  1.  No 
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r  2.  Yes 


Question  17 

If  you  answered  ‘Yes’  to  the  prior  question,  enter  the  number  on  on-line  courses  you  have 
taken. 

Answer: 


Question  18 

How  often  do  you  expect  to  use  this  course  web  site? 

r  1.  3-5  times  per  week, 
r  2.  2  times  a  week, 
r  3.  At  least  once  a  week, 
r  4.  I  don't  know. 

Question  19 

The  computer  you  plan  to  use  most  often  to  access  the  course  is  located: 

r  1.  InmyBOQroom. 
r  2.  In  my  house, 
r  3.  In  a  computer  lab  on  campus, 
r  4.  At  my  place  of  work, 
r  5.  In  the  library. 

c  6.  In  a  shared  area  of  my  living  quarters  (e.g.,  home,  BOQ,  apartment,  etc.), 
r  7.  Other  -  complete  following  question. 
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Question  20 

Please  type  in  location/s,  if  the  computer  you  plan  to  use  most  often  is  some  place  not 
listed  in  die  prior  question. 


Question  21 

On  a  scale  of  1  to  5  where  l=novice  and  5=expert,  how  would  you  rate  your  current 
proficiency  in  using  computers? 

r 

1.  1 

r 

2.  2 

r 

3.  3 

c 

4.  4 

r 

5.  5 

Question  22 

On  a  scale  of  1  to  5  where  l=novice  and  5=expert,  how  would  you  rate  your  on-line 
search  resources  and  techniques?  [Five-choice  scale,  from  1  to  5.] 

Question  23 

Please  indicate  the  extent  to  which  you  agree  with  the  following  statement: 
I  expect  this  on-line  course  will  be  valuable  and  improve  my  learning: 
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r  1.  Strongly  disagree. 

c  2.  Somewhat  disagree, 

r  3.  Neutral, 

r  4.  Somewhat  agree. 

c  5.  Strongly  agree. 


Question  24 

Please  indicate  the  extent  to  which  you  agree  with  the  following  statement: 

I  am  concerned  about  my  ability  to  use  the  web-based  materials  in  this  class:  [Five-choice 
scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  25 

Any  additional  comments,  concerns,  or  suggestions  are  welcome: 


APPENDIX  B.  A-2  QUESTIONNAIRE 


NUMBER  OF  QUESTIONS:  26 


You  Have  Completed  The  Orientation  Portion  Of  This  Course. 

How  Are  Things  Going  So  Far? 

Question  1 

How  would  you  rate  your  comfort  level  at  this  point  with  setting-up  your  system? 

r 

1.  Extremely  uncomfortable. 

r 

2.  Somewhat  uncomfortable. 

r 

3.  Neutral. 

r 

4.  Somewhat  comfortable. 

r 

5.  Extremely  comfortable. 

Question  2 

How  would  you  rate  your  comfort  level  at  this  point  with  accessing  the  Internet?  [Five- 
choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  3 

How  would  you  rate  your  comfort  level  at  this  point  with  word  processing?  [Five-choice 
scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  4 

How  would  you  rate  your  comfort  level  at  this  point  with  posting  to  the  forum?  [Five- 
choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  5 
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How  would  you  rate  your  comfort  level  at  this  point  with  sending  electronic  mail?  [Five- 
choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  6 

In  what  areas  of  the  on-line  technologies  (system  set-up,  accessing  the  internet,  word¬ 
processing,  posting  to  forums)  do  you  still  need  help? 


Question  7 

Please  provide  any  additional  thoughts. 


Question  8 

How  would  you  rate  your  comfort  level  at  this  point  with  the  technical  support  provided 
for  using  the  WebCT  software? 

r  1.  Extremely  uncomfortable, 

r  2.  Somewhat  uncomfortable. 

c  3.  Neutral, 

r  4.  Somewhat  comfortable. 

C  5.  Extremely  comfortable. 

Question  9 
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How  would  you  rate  your  comfort  level  at  this  point  with  the  technical  support  available 
to  you  through  your  Internet  service  provider  (ISP)?  [Five-choice  scale,  from  ‘Extremely 
uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  10 

How  would  you  rate  your  comfort  level  at  this  point  with  the  technical  support  available 
to  you  through  the  university  ADP  office?  [Five-choice  scale,  from  ‘Extremely 
uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  11 

How  would  you  rate  your  comfort  level  at  this  point  with  the  technical  support  available 
to  you  on  CD-ROM?  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely 
comfortable’.] 


Question  12 

What  technical  issues  (e.g.,  those  related  to  WebCT,  Internet  Services,  Local  ADP,  CD- 
ROM,  etc.)  do  you  still  need  to  resolve? 


Question  13 

Please  provide  any  additional  thoughts. 


279 


Question  14 


How  would  you  rate  your  comfort  level  at  this  point  with  the  forum  feature  being  used 
to  teach  this  course?  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely 
comfortable’.] 


Question  15 

How  would  you  rate  your  comfort  level  at  this  point  with  the  email  feature  being  used  to 
teach  this  course?  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely 
comfortable’.] 


Question  16 

How  would  you  rate  your  comfort  level  at  this  point  with  quiz  feature  being  used  to 
teach  this  course?  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely 
comfortable’.] 


Question  17 

How  would  you  rate  your  comfort  level  at  this  point  with  the  links  (i.e.,  additional  web 
sites)  feature  being  used  to  teach  this  course?  [Five-choice  scale,  from  ‘Extremely 
uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  18 


How  effective  was  the  course  overview?  That  is,  to  what  extent  did  the  course  overview 
help  you  to  understand  what  you  would  learn  in  the  ‘Space  Systems  -  Technologies  and 
Applications’  on-line  course.  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to 
‘Extremely  comfortable’.] 


Question  19 

How  effective  were  module  objectives?  That  is,  to  what  extent  did  the  module 
objectives  help  you  to  understand  what  you  would  learn  in  the  ‘Space  Systems  - 
Technologies  and  Applications’  on-line  course.  [Five-choice  scale,  from  ‘Extremely 
uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  20 

How  effective  were  the  rules  of  engagement?  That  is,  to  what  extent  did  the  rules  of 
engagement  help  you  to  understand  what  would  be  involved  in  taking  the  ‘Space 
Systems  -  Technologies  and  Applications’  on-line  course.  [Five-choice  scale,  from 
‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  21 

How  effective  was  the  course  syllabus?  That  is,  to  what  extent  did  the  course  syllabus 
help  you  to  understand  what  you  would  learn  in  the  ‘Space  Systems  -  Technologies  and 
Applications’  on-line  course.  [Five-choice  scale,  from  ‘Extremely  uncomfortable’  to 
‘Extremely  comfortable’.] 


Question  22 


In  what  areas  do  you  still  need  help  with  the  web-based  technological  features  (forum, 
email,  quizzes,  links,  etc)  used  to  teach  this  on-line  course? 
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Question  23 

So  far,  what  do  you  like  about  this  on-line  course? 


Question  24 

So  far,  what  are  your  concerns  about  this  on-line  course? 


Question  25 


Please  respond  to  the  following  question:  ‘If  I  were  building  an  on-line  course  to  teach 
space  systems-related  instruction,  I  would. . 


Question  26 

Do  you  have  any  suggestions,  at  this  point  (in  particular,  about  your  orientation  to  taking 
a  course  on-line)? 
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APPENDIX  C.  A-14  QUESTIONNAIRE 


Throughout  Module  14  you  developed  and  practiced  your  skills  in  the  process  of 
analyzing,  articulating,  and  evaluating  DoD  Space  Control  issues.  The  statements  below 
reflect  the  learning  objectives  for  Module  14.  To  what  extent  do  you  agree  that  the 
material  presented  in  the  Module  enabled  you  to  achieve  the  learning  objectives? 


Question  1 

Identifying  key  elements  of  Space  Control.  (Check  one  of  seven  points  that  fits  your 
opinion  best.) 

c  1.  Strongly  disagree, 

r  2.  Disagree, 

r  3.  Slightly  disagree, 

r  4.  Neutral. 

c  5.  Slightly  agree, 

r  6.  Agree. 

r  7.  Strongly  agree. 


Question  2 

Synthesizing  other's  findings  in  Forums.  [Seven-choice  scale,  from  ‘Strongly  disagree’  to 
‘Strongly  agree’.] 


Question  3 

Interacting  with  a  subject-matter  expert  whose  knowledge  you  can  use  to  solve  space- 
system  problems.  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 
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Question  4 


Participating  in  forums  (e.g.  discussing,  identifying/consulting  subject  matter  experts),  to 
evaluate  space  control  issues.  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly 
agree’.] 


Question  5 

Articulating  how  the  concept  of  space  control  relates  to  Joint  Vision  2020.  [Seven-choice 
scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  6 

Evaluating  the  likelihood  that  various  types  of  space  countermeasures  might  be  used 
across  the  peace- war  continuum.  [Seven-choice  scale,  from  ‘Strongly  disagree’  to 
‘Strongly  agree’.] 


Question  7 

Identifying  elements  of  space  systems  that  are  vulnerable  to  disruption,  degradation  or 
destruction  by  adversaries.  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly 
agree’.] 


Question  8 

In  Evaluation  A2  we  asked  you  several  questions  about  your  comfort  level  after  the 
orientation  part  of  the  course.  Now,  when  the  course  is  practically  over,  we  would  like 
you  to  reassess  some  of  these  issues,  i.e.  to  think  how  your  comfort  level  really  was 
throughout  the  course. 

How  would  you  rate  your  comfort  level  at  this  point  with  accessing  the  Internet?  (Check 
one  of  seven  possible  answers.) 


r  1.  Extremely  uncomfortable, 
r  2.  Uncomfortable. 
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r 

3. 

Somewhat  uncomfortable. 

r 

4. 

Neutral. 

r 

5. 

Somewhat  comfortable. 

r 

6. 

Comfortable. 

r 

7.  Extremely  comfortable. 

Question  9 

How  would  you  rate  your  comfort  level  at  this  point  with  sending  electronic  mail? 
[Seven-choice  scale,  from  ‘Extremely  uncomfortable’  to  ‘Extremely  comfortable’.] 


Question  10 

How  would  you  rate  your  comfort  level  at  this  point  with  posting  comments  to  the 
Forums?  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  11 

How  would  you  rate  your  comfort  level  at  this  point  with  quiz  feature  being  used  to  teach 
this  course?  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  12 

How  would  you  rate  your  comfort  level  at  this  point  with  the  links  (i.e.,  additional  web 
sites)  feature  being  used  to  teach  this  course?  [Seven-choice  scale,  from  ‘Strongly 
disagree’  to  ‘Strongly  agree’.] 


Question  13 

How  would  you  rate  your  comfort  level  at  this  point  with  word  processing?  [Seven- 
choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  14 

Please  provide  any  additional  thoughts. 
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APPENDIX  D.  FINAL  SURVEY  QUESTIONNAIRE 


Number  of  Questions:  31 


Please  answer  the  following  questions  as  honestly  as  you  can.  The  information  will  be 
compiled  and  evaluated  by  a  thesis  student  and  will  serve  as  NPS's  foundation  for 
building  future  on-line  courses.  Thanks  for  taking  the  time  to  participate. 

CDR  Sue  Higgins 


Question  1 

As  a  result  of  your  overall  experience  in  this  course,  do  you  feel  that  you  are  able  to 
articulate  how  physical  influences  on  space  systems  impact  our  ability  to  use  those 
systems  in  military  operational  situations? 

r  1.  Strongly  disagree, 

r  2.  Disagree, 

r  3.  Slightly  disagree, 

r  4.  Neutral. 

r  5.  Slightly  agree, 

r  6.  Agree, 

r  7.  Strongly  agree. 


Question  2 

As  a  result  of  your  experiences  in  this  course,  are  you  able  to  articulate  how  political, 
organizational  and  economic  influences  impact  how  we  use  space  systems  in  military 
operational  settings?  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 


Question  3 

Do  you  feel  that  you  are  able  to  articulate  how  commercial  space  systems  will  impact 
military  operations?  [Seven-choice  scale,  from  ‘Strongly  disagree’  to  ‘Strongly  agree’.] 
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Question  4 

Was  the  final  exam  useful  in  helping  you  to  synthesize  the  concepts  learned  in  the 
course? 

r 

1.  Extremely  useless. 

r 

2.  Useless. 

r 

3.  Slightly  useless. 

r 

4.  Neutral. 

r 

5.  Slightly  useful. 

r 

6.  Useful. 

r 

7.  Extremely  useful. 

Question  5 

I  feel  confident  in  my  ability  to  access  appropriate  resources,  such  as  subject  matter 
experts,  documentation  of  websites,  or  late  breaking  news  for  evaluating  uses  of  space 
systems  in  operational  situations. 

r 

1.  Strongly  disagree. 

r 

2.  Disagree. 

r 

3.  Slightly  disagree. 

r 

4.  Neutral. 

r 

5.  Slightly  agree. 

r 

6.  Agree. 

r 

7.  Strongly  agree. 

Question  6 

I  feel  confident  in  my  ability  to  get  involved  in  open  discussions  with  my  peers  to  weigh 
issues  related  to  using  space  systems  in  operational  settings.  [Seven-choice  scale,  from 
‘Strongly  disagree’  to  ‘Strongly  agree’.] 
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Question  7 


How  useful  were  the  guidelines  provided  in  the  course  homepage  to  your  ability  to 
succeed? 


r  1.  Extremely  useless, 
r  2.  Useless. 

r  3.  Slightly  useless. 

r  4.  Neutral. 

r  5.  Slightly  useful. 

c  6.  Useful. 

c  7.  Extremely  useful. 


Question  8 

How  useful  were  guidelines  provided  at  the  course  homepage  for  posting  substantive 
responses  to  the  Forums?  [Seven-choice  scale,  from  ‘Extremely  useless’  to  ‘Extremely 
useful’.] 


Question  9 

How  useful  was  the  textbook  in  helping  you  understand  the  course  concepts,  objectives 
and  principles?  [Seven-choice  scale,  from  ‘Extremely  useless’  to  ‘Extremely  useful’.] 


Question  10 

How  useful  were  the  on-line  Module  readings  in  helping  you  to  understand  the  course 
concepts,  objectives  and  principles?  [Seven-choice  scale,  from  ‘Extremely  useless’  to 
‘Extremely  useful’.] 


Question  11 

How  useful  were  the  linked  websites  in  helping  you  to  understand  the  course  concepts, 
objectives  and  principles?  [Seven-choice  scale,  from  ‘Extremely  useless’  to  ‘Extremely 
useful’.] 
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Question  12 

How  useful  were  the  Forum  exercises  in  helping  you  to  understand  the  subject  matter? 
[Seven-choice  scale,  from  ‘Extremely  useless’  to  ‘Extremely  useful’.] 


Question  13 

How  useful  were  the  Quizzes  (2,  3/4/5,  6/7)  for  providing  feedback  in  your  understanding 
of  the  material  for  those  Modules?  (Try  to  separate  the  technical  problems  with  the  early 
quiz  from  your  answer.)  [Seven-choice  scale,  from  ‘Extremely  useless’  to  ‘Extremely 
useful’.] 


Question  14 

How  did  you  like  the  Forum  assignments? 

r  1.  Strongly  disliked, 

r  2.  Disliked, 

r  3.  Slightly  disliked, 

r  4.  Neutral. 

c  5.  Slightly  liked, 

r  6.  Liked, 

r  7.  Strongly  liked. 

Question  15 

How  did  you  like  the  textbook  reading  assignments?  [Seven-choice  scale,  from  ‘Strongly 
disliked’  to  ‘Strongly  liked’.] 


Question  16 

How  did  you  like  the  Module  readings?  [Seven-choice  scale,  from  ‘Strongly  disliked’  to 
‘Strongly  liked’.] 
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Question  17 


How  did  you  like  the  website  links  that  were  provided?  [Seven-choice  scale,  from 
‘Strongly  disliked’  to  ‘Strongly  liked’.] 


Question  18 

What  other  instructional  materials,  if  any,  should  be  included  in  the  course? 


Question  19 

Describe  your  reaction  to  the  instructional  strategies  used  to  teach  the  subject  matter 
(textbook  readings,  forums,  websites). 


Question  20 

I  am  certain  that  I  am  able  to  synthesize  information  from  a  variety  of  sources  to  enable 
me  to  use  space  systems  in  operational  situations. 
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r  1.  Strongly  disagree, 

r  2.  Disagree. 

r  3.  Slightly  disagree, 

r  4.  Neutral, 

r  5.  Slightly  agree, 

r  6.  Agree, 

r  7.  Strongly  agree. 


Question  21 

As  a  result  of  this  on-line  course  experience,  do  you  feel  that  you  have  changed  your 
fundamental  approach  to  learning?  Please  respond  ‘yes’,  ‘somewhat’  or  ‘no’,  and  explain 
your  answer. 


Question  22 

Did  you  view  the  flexibility  of  this  on-line  course  as  beneficial  to  your  ability  to  succeed 
in  this  class? 


r  1.  Strongly  disagree. 

r  2.  Disagree. 
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r  3.  Slightly  disagree. 
r  4.  Neutral, 

r  5.  Slightly  agree, 

r  6.  Agree, 

r  7.  Strongly  agree. 


Question  23 

Do  you  feel  that  the  flexibility  of  this  class  helped  you  succeed  in  other  classes  that  you 
were  taking  in  the  same  quarter?  [Seven-choice  scale,  from  ‘Strongly  disagree’  to 
‘Strongly  agree’.] 


Question  24 

Compare  the  amount  of  interaction  you  had  with  the  instructor  for  this  on-line  course  as 
compared  with  a  similar  resident  ‘face-to-face’  course  (note  -  this  question  is  about 
‘amount’;  another  question  will  cover  ‘quality  of  interaction’).  Try  to  consider  only  the 
amount  of  interaction  that  occurred  on-line  (including  e-mails)-  as  opposed  to  what 
occurred  if/when  you  stopped  by  the  instructor's  office. 

In  this  on-line  class  I  had  significantly  less  interaction  with  the  instructor  than  I 
would  have  in  a  similar  face-to-face  class. 

In  this  on-line  class  I  had  less  interaction  with  the  instructor  than  I  would  have  in 
a  similar  face-to-face  class. 

In  this  on-line  class  I  had  slightly  less  interaction  with  the  instructor  than  I  would 
have  in  a  similar  face-to-face  class. 

The  amount  of  interaction  between  the  instructor  and  me  was  about  the  same. 

In  this  on-line  class  I  had  slightly  more  interaction  with  the  instructor  than  I 
would  have  in  a  similar  face-to-face  class. 

In  this  on-line  class  I  had  more  interaction  with  the  instructor  than  I  would  have 
in  a  similar  face-to-face  class. 

In  this  on-line  class  I  had  significantly  more  interaction  with  the  instructor  than  I 
would  have  in  a  similar  face-to-face  class. 


1. 

2. 

3. 

4. 

5. 

r  6. 

7. 


Question  25 

Compare  the  quality  of  interaction  between  yourself  and  the  instructor  in  this  on-line 
class  against  what  you  would  have  expected  in  a  similar  "face-to-face"  class. 
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r  j  In  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 
‘  was  of  significantly  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  2  h1  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 
‘  was  of  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  ^  &1  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 
'  was  of  slightly  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  4.  The  quality  of  interaction  between  the  instructor  and  myself  was  about  the  same. 

r  ^  In  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 

'  was  of  slightly  greater  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  ^  In  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 
‘  was  of  greater  quality  than  it  would  have  been  in  a  face-to-face  class. 

In  this  on-line  course  the  quality  of  interaction  between  the  instructor  and  myself 

c  7.  was  of  significantly  greater  quality  than  it  would  have  been  in  a  face-to-face 
class. 


Question  26 

Compare  the  amount  of  interaction  you  had  with  other  students  for  this  on-line  course  as 
compared  with  a  similar  resident  ‘face-to-face’  course  (note  -  this  question  is  about 
‘amount’;  another  question  will  cover  ‘quality  of  interaction’.) 


r  i  In  this  on-line  course  I  had  significantly  less  interaction  with  other  students  than 
'  in  a  similar  face-to-face  course. 

r  2  1°  this  on-line  course  I  had  less  interaction  with  other  students  than  in  a  similar 
'  face-to-face  course. 

r  ^  I*1  this  on-line  course  I  had  slightly  less  interaction  with  other  students  than  in  a 
similar  face-to-face  course. 

r  4.  The  amount  of  interaction  was  about  the  same. 

r  ^  In  this  on-line  course  I  had  slightly  more  interaction  with  other  students  than  in  a 
similar  face-to-face  course. 

r  g  In  this  on-line  course  I  had  more  interaction  with  other  students  than  in  a  similar 
‘  face-to-face  course. 

r  ^  I11  this  on-line  course  I  had  significantly  more  interaction  with  other  students 
‘  than  in  a  similar  face-to  face  course. 


Question  27 
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Compare  the  quality  of  interaction  you  had  with  other  students  for  this  on-line  course  as 
compared  with  a  similar  resident  ‘face-to-face’  course. 


r  i  In  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
'  was  of  significantly  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  ^  Ih  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
‘  was  of  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  ^  In  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
'  was  of  slightly  less  quality  than  it  would  have  been  in  a  face-to-face  class. 

r  4.  The  quality  of  interaction  was  about  the  same. 


r 

r 


r 


^  In  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
'  was  of  slightly  greater  quality  than  it  would  have  been  in  a  face-to-face  class. 

6  In  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
'  was  of  greater  quality  than  it  would  have  been  in  a  face-to-face  class. 

In  this  on-line  course  the  quality  of  interaction  between  myself  and  other  students 
7.  was  of  significantly  greater  quality  than  it  would  have  been  in  a  face-to-face 
class. 


Question  28 

Describe  your  reaction  to  the  media  used  to  present  the  instruction  (i.e.  online  delivery 
instructional  technologies.) 


Question  29 

If  you  wish  to  add  any  other  comments  about  the  course,  use  this  space. 
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Question  30  (Copyright  1998  ©  William  A.  Long  and  Charles  D.  Dziuban) 

This  question  will  try  to  capture  your  personal  traits  that  are  related  to  your  on-line 
learning  style.  These  questions  are  very  easy  to  answer,  but  may  take  some  effort  to  think 
honestly  about  yourself.  We  stress  that  your  responses  will  be  used  anonymously  for 
research  purposes  only.  The  following  sets  represent  main  types  of  on-line  learning 
styles.  The  types  are  not  what  people  think  they  should  be  like,  or  what  sounds  like  a 
desirable  description,  but  rather  as  people  really  are.  Choose  what  was  closest  to  you  as 
you  went  through  this  on-line  course. 

Highly  energized  and  action-oriented.  Little  need  for  approval;  unconcerned  with 
r  1.  who  they  please.  Put  thinking  into  immediate  action.  Very  frank,  speaks  out 
freely.  Has  no  problems  confronting  people. 

Lower  energy  level.  Little  need  for  approval  -  unconcerned  with  pleasing  others. 
r  2.  Independent  &  strong-willed.  Prefers  to  work  alone.  May  resist  pressure  from 
authority.  Independent  thinker. 

Highly  energized  and  productive.  Strongly  motivated  by  approval.  Sensitive  to 
the  wishes  of  others.  Translates  energies  into  constructive  tasks.  Deeply  values 
r  3.  close  bonds  with  others.  Some  difficulty  dealing  with  direct  confrontation. 
Highly  idealistic,  setting  lofty  goals  for  themselves.  Foster  harmonious 
relationships. 

Lower  energy  level.  Needs  approval  -  concerned  with  pleasing  others.  Rarely 
r  4.  shows  anger  or  resentment.  Very  sensitive  to  the  feelings  of  others.  Very 
compliant  and  loyal.  Forms  strong  attachments.  Gives  and  thrives  on  affection. 


Question  31  (Copyright  1998  ©William  A.  Long  and  Charles  D.  Dziuban) 

Please  read  carefully  the  descriptions  in  the  four  boxes  below.  Although  not  all  behaviors 
may  fit  you  exactly,  select  as  many  boxes  that  you  feel  apply  to  you.  Here  you  may  pick 
from  none  to  four  boxes. 
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j  Thinks  of  all  possibilities  and  contingencies  before  venturing  into  activities. 
‘What  if  person.  May  see  the  negative  side  of  things.  Unwilling  to  take  risks. 

Highly  organized  and  methodical.  Strongly  motivated  to  finish  tasks. 
2.  Perfectionist.  Tends  to  form  habits.  Extremely  diligent  in  work  habits.  May  be 
mildly  ritualistic. 

3  Sometimes  explosive  and  quick  tempered.  Sharp  tongued.  Very  frank.  May  act 
'  without  thinking. 

4  Dramatic.  May  have  wide  mood  swings.  May  overreact  in  some  situations.  Can 
'  have  emotional  outbursts.  Artistically  inclined.  Devalues  work  routine. 
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APPENDIX  E.  INTERVIEWS 


INTRODUCTION 

SS3011  on-line  course  has  finished  just  a  couple  of  months 
ago.  Now  you  have  nice  time  distance  from  the  course,  so  your 
impressions  and  opinion  about  it  are  more  clear  and  final.  The 
purpose  of  this  brief  interview  is  to  try  to  capture  some  of  your 
deeper  insights  and  conclusions  about  your  experience  of  that 
course. 

As  in  all  four  on-line  evaluation  questionnaires  you 
answered  during  the  course,  your  answers  will  be  used  only  for  my 
thesis  research  and  your  identity  will  be  protected.  Please  answer 
them  straightforward  and  comment  as  you  wish. 


QUESTIONS 

1.  In  general,  could  you  estimate  have  you  learned  more  from  the  on-line  designed 
course  than  you  would  learn  from  a  face-to-face  SS3011  course?  I’m  asking  you 
about  successful  learning  of  subject  matter  taught  through  distributed  learning, 
not  about  particular  instructor  or  anything  else. 

2.  Students’  answers  about  forums  exercises  were  very  different,  in  terms  of  forum 
exercises  being  useful  in  helping  to  understand  the  subject  matter.  What  is  your 
final  opinion  about  their  real  usefulness  in  facilitating  your  learning?  Also, 
comment  how  much  you  liked  using  forums. 

3.  The  same  question  about  Quizzes  -  please  comment  how  useful  they  were  for 
you  and  how  much  you  liked  them! 

4.  Students’  answers  to  question  about  how  they  saw  the  amount  of  interaction 
with  the  instructor  during  the  SS3011  course,  compared  to  a  similar  resident 
(face-to-face)  course,  were  very  different.  Please  comment  your  view. 
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5.  I  want  to  ask  you  similar  question  about  the  quality  of  students’  interaction 
with  the  instructor,  compared  to  a  resident  course.  How  you  saw  it? 

6.  One  of  most  controversial  questions,  according  to  students’  answers,  was  about 
the  amount  of  interaction  between  the  students,  compared  to  a  face-to-face 
course.  Some  thought  SS3011  offered  significantly  less  interaction,  some  others  - 
significantly  more.  What’s  your  comment? 

7.  The  last  question  about  interactions  covers  the  issue  of  quality  of  interaction 
between  the  students,  what  is  important  for  the  learning  process.  How  would  you 
comment  it,  in  sense  of  comparing  the  quality  of  interaction  between  students  in 
on-line  SS3011  with  interaction  between  students  in  some  similar  face-to-face 
course? 

8.  If  you  have  an  opportunity  to  choose  between  on-line  and  face-to-face  section  of 
some  other  class  in  your  curriculum,  what  would  you  do?  (Examples:  I  would 
surely  go  online;  depends  on  nature  of  the  course;  depends  on  my  schedule; 
surely  stay  in  face-to-face  section...) 

9.  Please  tell  us  any  advice/recommendation/suggestion  you  have  for  CDR 
Higgins  or  any  other  NPS  or  Navy  distributed  learning  instructor.  (This  question 
is  very  open-ended!) 

10.  This  course  was  not  only  a  space-systems  course,  but  also  a  new  experience  for 
most  of  the  students  in  the  section.  Looking  at  the  course  from  this  time  distance 
and  in  general,  do  you  think  you’ve  gained  some  other  benefits  besides  the 
course  objectives  themselves?  Please  comment. 

11. 1  asked  you  really  many  questions  in  the  four  online  questionnaires  and  in  this 
interview.  If  you  think  there’s  something  else  I  should  know,  but  you  haven’t 
comment  it  yet,  please  feel  free  to  do  it  now. 

At  the  end,  I  just  want  to  express  my  appreciation  for  your  time,  patience  and 
efforts.  I  will  do  my  best  to  make  it  maximally  beneficial  for  future  students,  instructors 
and  NPS  in  general,  what  your  answers  made  possible.  Thank  you! 
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